TY - JOUR T1 - Expert-validated estimation of diagnostic uncertainty for deep neural networks in diabetic retinopathy detection JF - medRxiv DO - 10.1101/19002154 SP - 19002154 AU - Murat Seçkin Ayhan AU - Laura Kühlewein AU - Gulnar Aliyeva AU - Werner Inhoffen AU - Focke Ziemssen AU - Philipp Berens Y1 - 2019/01/01 UR - http://medrxiv.org/content/early/2019/07/15/19002154.abstract N2 - Deep learning-based systems can achieve a diagnostic performance comparable to physicians in a variety of medical use cases including the diagnosis of diabetic retinopathy. To be useful in clinical practise, it is necessary to have well calibrated measures of the uncertainty with which these systems report their decisions. However, deep neural networks (DNNs) are being often overconfident in their predictions, and are not amenable to a straightforward probabilistic treatment. Here, we describe an intuitive framework based on test-time data augmentation for quantifying the diagnostic uncertainty of a state-of-the-art DNN for diagnosing diabetic retinopathy. We show that the derived measure of uncertainty is well-calibrated and that experienced physicians likewise find cases with uncertain diagnosis difficult to evaluate. This paves the way for an integrated treatment of uncertainty in DNN-based diagnostic systems.Competing Interest StatementThe authors have declared no competing interest.Funding StatementThis research was supported by the German Ministry of Science and Education (BMBF, 01GQ1601 and 01IS18039A) and the German Science Foundation (BE5601/4-1 and EXC 2064, project number 390727645).Author DeclarationsAll relevant ethical guidelines have been followed and any necessary IRB and/or ethics committee approvals have been obtained.Not ApplicableAll necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived.Not ApplicableAny clinical trials involved have been registered with an ICMJE-approved registry such as ClinicalTrials.gov and the trial ID is included in the manuscript.Not ApplicableI have followed all appropriate research reporting guidelines and uploaded the relevant Equator, ICMJE or other checklist(s) as supplementary files, if applicable.Not ApplicableAll data used in this study are publicly available. More information can be found via the following links: Dataset 1: https://www.kaggle.com/c/diabetic-retinopathy-detection Dataset 2: https://idrid.grand-challenge.org/ ER -