ABSTRACT
BACKGROUND Predicting outcomes in schizophrenia spectrum disorders is challenging due to the variability of individual trajectories. While machine learning (ML) shows promise in outcome prediction, is has not yet been integrated into clinical practice. Understanding how ML models (MLMs) can complement psychiatrists’ predictions and bridge the gap between MLM capabilities and practical use is key.
OBJECTIVE This study aims to compare the performance of psychiatrists and MLMs in predicting short-term symptomatic and functional remission in patients with first-episode psychosis and explore whether MLMs can improve psychiatrists’ prognostic accuracy.
METHOD Twenty-four psychiatrists predicted symptomatic and functional remission probabilities based on written baseline information from 66 patients in the OPTiMiSE trial. ML-generated predictions were then shared with psychiatrists, allowing them to adjust their estimates. A questionnaire assessed trust in MLMs, perceived information gaps, and psychiatrists’ self-assessed predictive accuracy, which was compared to actual accuracy.
FINDINGS The predictive accuracy of the MLM was comparable to that of psychiatrists for symptomatic remission (MLM: 0.50, psychiatrists: 0.52) and functional remission (MLM: 0.72, psychiatrists: 0.79). Interrater agreement was low but comparable for psychiatrists and the MLM. Although the MLM did not improve overall predictive accuracy, it showed potential in aiding psychiatrists with difficult-to-predict cases. However, psychiatrists struggled to recognize when to rely on the model’s output and we were unable to determine a clear pattern in these cases based on their characteristics. Psychiatrists could not reliably estimate their predictive accuracy. Psychiatrists expressed moderate to high trust in MLMs for prognostic prediction, but highlighted concerns about the lack of transparency and interpretability of model outputs.
CONCLUSIONS MLMs are a promising tool for supporting psychiatric decision-making, particularly in challenging cases. However, their potential remains underutilized due to limitations in predictive accuracy and a lack of clarity in how predictions are generated. Addressing these issues is essential to build trust and foster integration into clinical practice.
CLINICAL IMPLICATIONS MLMs are best suited as supplementary tools, providing a second opinion while psychiatrists retain decision-making autonomy. Integrating predictions from both sources may help reduce individual biases and improve accuracy. This approach leverages the strengths of MLMs without compromising clinical responsibility.
What is already known on this topic While machine learning models (MLMs) show promise in predicting outcomes in psychotic disorders, they have yet to be integrated into clinical practice. Evidence on the predictive accuracy of psychiatrists for these disorders is limited, with only two small studies published before 1990 suggesting moderate accuracy. Comparisons of MLMs and psychiatrists in this context have not been previously conducted.
What this study adds This is the first study to compare the predictive accuracy of psychiatrists with that of an MLM for psychotic disorders and to assess whether an MLM can enhance psychiatrists’ performance. It highlights that while MLMs do not improve overall accuracy, they may support psychiatrists in difficult cases. Insights into psychiatrists’ trust in MLMs and the challenges of implementing these models are also provided.
How this study might affect research, practice, or policy The findings emphasize the need for advancements in MLM accuracy, interpretability, and strategies to identify cases where MLMs are most beneficial. These improvements could foster effective integration of MLMs as supplementary tools in clinical practice, aiding psychiatrists in decision-making while maintaining their autonomy.
Competing Interest Statement
The authors have declared no competing interest.
Funding Statement
This work was supported by ZonMw (project ID 63631 0011) and by a grant from the working group AI for Health of the Alliance TU/eWUR-UU-UMCU for the project.
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
The details of the IRB/oversight body that provided approval or exemption for the research described are given below:
All procedures involving human subjects were approved by the Ethics Committee of the University Medical Centre Utrecht (protocol number 20/053).
I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.
Yes
