ABSTRACT
Classifying subjects with psychiatric diseases based on structural magnetic resonance imaging (MRI) data is a highly challenging task. Nevertheless, a number of studies report decoding accuracies of up to 90% using relatively small sample sizes. Here, we will present contradicting results on a large scale (N = 2240) dataset of subjects with and without depression as part of our participation in the Predictive Analytics Competition (PAC) 2018, in which we achieved the 3rd place. Contrary to our expectations, classification accuracies varied only little between a variety of simple and highly-complex classifiers and did not exceed 67% based on an internal validation set. We discuss our results in light of these opposing results and conclude that well designed challenges and large sample sizes provide a good way to get unbiased estimates of prediction performance.
Competing Interest Statement
The authors have declared no competing interest.
Funding Statement
We acknowledge support from the German Research Foundation (KR: DFG, 389563835), the Manfred and Ursula-M?ller Stiftung (KR), Charit? ? Universit?tsmedizin Berlin (KR: Rahel-Hirsch scholarship) and Studienstiftung des Deutschen Volkes (LD).
Author Declarations
All relevant ethical guidelines have been followed and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
All necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived.
Yes
Any clinical trials involved have been registered with an ICMJE-approved registry such as ClinicalTrials.gov and the trial ID is included in the manuscript.
NA
I have followed all appropriate research reporting guidelines and uploaded the relevant Equator, ICMJE or other checklist(s) as supplementary files, if applicable.
NA
Data Availability
Data used in this study were collected from different sites and were only released for the purpose of the PAC challenge.
