TY - JOUR T1 - Limitations of machine learning in psychiatry: Participation in the PAC 2018 depression challenge JF - medRxiv DO - 10.1101/19000562 SP - 19000562 AU - Fabian Eitel AU - Sebastian Stober AU - Lea Waller AU - Lena Dorfschmidt AU - Henrik Walter AU - Kerstin Ritter Y1 - 2019/01/01 UR - http://medrxiv.org/content/early/2019/06/25/19000562.abstract N2 - Classifying subjects with psychiatric diseases based on structural magnetic resonance imaging (MRI) data is a highly challenging task. Nevertheless, a number of studies report decoding accuracies of up to 90% using relatively small sample sizes. Here, we will present contradicting results on a large scale (N = 2240) dataset of subjects with and without depression as part of our participation in the Predictive Analytics Competition (PAC) 2018, in which we achieved the 3rd place. Contrary to our expectations, classification accuracies varied only little between a variety of simple and highly-complex classifiers and did not exceed 67% based on an internal validation set. We discuss our results in light of these opposing results and conclude that well designed challenges and large sample sizes provide a good way to get unbiased estimates of prediction performance.Competing Interest StatementThe authors have declared no competing interest.Funding StatementWe acknowledge support from the German Research Foundation (KR: DFG, 389563835), the Manfred and Ursula-M?ller Stiftung (KR), Charit? ? Universit?tsmedizin Berlin (KR: Rahel-Hirsch scholarship) and Studienstiftung des Deutschen Volkes (LD).Author DeclarationsAll relevant ethical guidelines have been followed and any necessary IRB and/or ethics committee approvals have been obtained.YesAll necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived.YesAny clinical trials involved have been registered with an ICMJE-approved registry such as ClinicalTrials.gov and the trial ID is included in the manuscript.NAI have followed all appropriate research reporting guidelines and uploaded the relevant Equator, ICMJE or other checklist(s) as supplementary files, if applicable.NAData used in this study were collected from different sites and were only released for the purpose of the PAC challenge.https://www.photon-ai.com/pac ER -