PT  - JOURNAL ARTICLE
AU  - Steven A. Hicks
AU  - Inga Strümke
AU  - Vajira Thambawita
AU  - Malek Hammou
AU  - Michael A. Riegler
AU  - Pål Halvorsen
AU  - Sravanthi Parasa
TI  - On evaluation metrics for medical applications of artificial intelligence
AID  - 10.1101/2021.04.07.21254975
DP  - 2021 Jan 01
TA  - medRxiv
PG  - 2021.04.07.21254975
4099  - http://medrxiv.org/content/early/2021/04/09/2021.04.07.21254975.short
4100  - http://medrxiv.org/content/early/2021/04/09/2021.04.07.21254975.full
AB  - Clinicians and model developers need to understand how proposed machine learning (ML) models could improve patient care. In fact, no single metric captures all the desirable properties of a model and several metrics are typically reported to summarize a model’s performance. Unfortunately, these measures are not easily understandable by many clinicians. Moreover, comparison of models across studies in an objective manner is challenging, and no tool exists to compare models using the same performance metrics. This paper looks at previous ML studies done in gastroenterology, provides an explanation of what different metrics mean in the context of the presented studies, and gives a thorough explanation of how different metrics should be interpreted. We also release an open source web-based tool that may be used to aid in calculating the most relevant metrics presented in this paper so that other researchers and clinicians may easily incorporate them into their research.Competing Interest StatementPal Halvorsen: Board member of Augere Medical Sravanthi Parasa: Consultant Covidien LP; Medical advisory board of FujifilmsFunding StatementNo external funding was receivedAuthor DeclarationsI confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.YesThe details of the IRB/oversight body that provided approval or exemption for the research described are given below:Not applicableAll necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived.YesI understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).YesI have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.YesNot applicable