PT - JOURNAL ARTICLE AU - Jenna M. Reps AU - Chungsoo Kim AU - Ross D. Williams AU - Aniek F. Markus AU - Cynthia Yang AU - Talita Duarte Salles AU - Thomas Falconer AU - Jitendra Jonnagaddala AU - Andrew Williams AU - Sergio Fernández-Bertolín AU - Scott L. DuVall AU - Kristin Kostka AU - Gowtham Rao AU - Azza Shoaibi AU - Anna Ostropolets AU - Matthew E Spotnitz AU - Lin Zhang AU - Paula Casajust AU - Ewout W. Steyerberg AU - Fredrik Nyberg AU - Benjamin Skov Kaas-Hansen AU - Young Hwa Choi AU - Daniel Morales AU - Siaw-Teng Liaw AU - Maria Tereza Fernandes Abrahão AU - Carlos Areia AU - Michael E. Matheny AU - María Aragón AU - Rae Woong Park AU - George Hripcsak AU - Christian G. Reich AU - Marc A. Suchard AU - Seng Chan You AU - Patrick B. Ryan AU - Daniel Prieto-Alhambra AU - Peter R. Rijnbeek TI - Can we trust the prediction model? Demonstrating the importance of external validation by investigating the COVID-19 Vulnerability (C-19) Index across an international network of observational healthcare datasets AID - 10.1101/2020.06.15.20130328 DP - 2020 Jan 01 TA - medRxiv PG - 2020.06.15.20130328 4099 - http://medrxiv.org/content/early/2020/06/17/2020.06.15.20130328.short 4100 - http://medrxiv.org/content/early/2020/06/17/2020.06.15.20130328.full AB - Background SARS-CoV-2 is straining healthcare systems globally. The burden on hospitals during the pandemic could be reduced by implementing prediction models that can discriminate between patients requiring hospitalization and those who do not. The COVID-19 vulnerability (C-19) index, a model that predicts which patients will be admitted to hospital for treatment of pneumonia or pneumonia proxies, has been developed and proposed as a valuable tool for decision making during the pandemic. However, the model is at high risk of bias according to the Prediction model Risk Of Bias ASsessment Tool and has not been externally validated.Methods We followed the OHDSI framework for external validation to assess the reliability of the C-19 model. We evaluated the model on two different target populations: i) 41,381 patients that have SARS-CoV-2 at an outpatient or emergency room visit and ii) 9,429,285 patients that have influenza or related symptoms during an outpatient or emergency room visit, to predict their risk of hospitalization with pneumonia during the following 0 to 30 days. In total we validated the model across a network of 14 databases spanning the US, Europe, Australia and Asia.Findings The internal validation performance of the C-19 index was a c-statistic of 0.73 and calibration was not reported by the authors. When we externally validated it by transporting it to SARS-CoV-2 data the model obtained c-statistics of 0.36, 0.53 (0.473-0.584) and 0.56 (0.488-0.636) on Spanish, US and South Korean datasets respectively. The calibration was poor with the model under-estimating risk. When validated on 12 datasets containing influenza patients across the OHDSI network the c-statistics ranged between 0.40-0.68.Interpretation The results show that the discriminative performance of the C-19 model is low for influenza cohorts, and even worse amongst COVID-19 patients in the US, Spain and South Korea. These results suggest that C-19 should not be used to aid decision making during the COVID-19 pandemic. Our findings highlight the importance of performing external validation across a range of settings, especially when a prediction model is being extrapolated to a different population. In the field of prediction, extensive validation is required to create appropriate trust in a model.Competing Interest StatementDr. Prieto-Alhambra reports grants and other from AMGEN, grants, non-financial support and other from UCB Biopharma, grants from Les Laboratoires Servier, outside the submitted work; and Janssen, on behalf of IMI-funded EHDEN and EMIF consortiums, and Synapse Management Partners have supported training programmes organised by DPA's department and open for external participants. Dr. Rijnbeek reports grants from Innovative Medicines Initiative, grants from Janssen Research and Development, during the conduct of the study. Dr. Reich and Ms. Kostka report they are employees of IQVIA. Dr. Reps, Dr. Ryan, Dr. Shoaibi and Dr. Rao are employees with compensation at Janssen Research & Development, JNJ. Dr. Suchard reports grants from US National Institutes of Health, grants from IQVIA, personal fees from Janssen Research and Development, personal fees from Private Health Management, during the conduct of the study. Dr. Morales is supported by a Wellcome Trust Clinical Research Development Fellowship (Grant 214588/Z/18/Z) and reports grants from Chief Scientist Office (CSO), grants from Health Data Research UK (HDR-UK), grants from National Institute of Health Research (NIHR), outside the submitted work. Dr. Hripcsak reports grants from US NIH National Library of Medicine, during the conduct of the study; grants from Janssen Research, outside the submitted work. Benjamin Skov Kaas-Hansen reports grants from Innovation Fund Denmark and Novo Nordisk Foundation, outside the submitted work. Funding StatementThis project has received support from the European Health Data and Evidence Network (EHDEN) project. EHDEN received funding from the Innovative Medicines Initiative 2 Joint Undertaking (JU) under grant agreement No 806968. The JU receives support from the European Union's Horizon 2020 research and innovation programme and EFPIA. This work was also supported by the Bio Industrial Strategic Technology Development Program (20001234, 20003883) funded by the Ministry of Trade, Industry & Energy (MOTIE, Korea) and a grant from the Korea Health Technology R&D Project through the Korea Health Industry Development Institute (KHIDI), funded by the Ministry of Health & Welfare, Republic of Korea [grant number: HI16C0992]. This project is funded by the Health Department from the Generalitat de Catalunya with a grant for research projects on SARS-CoV-2 and COVID-19 disease organized by the Direccio General de Recerca i Innovacio en Salut. The University of Oxford received a grant related to this work from the Bill & Melinda Gates Foundation (Investment ID INV-016201), and partial support from the UK National Institute for Health Research (NIHR) Oxford Biomedical Research Centre. DPA is funded through a NIHR Senior Research Fellowship (Grant number SRF-2018-11-ST2-004). The views expressed in this publication are those of the author(s) and not necessarily those of the NHS, the National Institute for Health Research or the Department of Health. BSKH is funded through Innovation Fund Denmark (5153-00002B) and the Novo Nordisk Foundation (NNF14CC0001). This project is part funded by the UNSW RIS grant.Author DeclarationsI confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.YesThe details of the IRB/oversight body that provided approval or exemption for the research described are given below:All databases obtained IRB approval or used deidentified data that was considered exempt from IRB approval. Informed consent was not necessary at any site. All necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived.All necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived.YesI understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).Yes I have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.YesData is not publicly available due to patient privacy concerns.