%0 Journal Article %A Brandon C. Cummings %A Sardar Ansari %A Jonathan R. Motyka %A Guan Wang %A Richard P. Medlin, Jr. %A Steven L. Kronick %A Karandeep Singh %A Pauline K. Park %A Lena M. Napolitano %A Robert P. Dickson %A Michael R. Mathis %A Michael W. Sjoding %A Andrew J. Admon %A Kevin R. Ward %A Christopher E. Gillies %T Validation and comparison of PICTURE analytic and Epic Deterioration Index for COVID-19 %D 2020 %R 10.1101/2020.07.08.20145078 %J medRxiv %P 2020.07.08.20145078 %X Introduction The 2019 coronavirus (COVID-19) has led to unprecedented strain on healthcare facilities across the United States. Accurately identifying patients at an increased risk of deterioration may help hospitals manage their resources while improving the quality of patient care. Here we present the results of an analytical model, PICTURE (Predicting Intensive Care Transfers and Other UnfoReseen Events), to identify patients at a high risk for imminent intensive care unit (ICU) transfer, respiratory failure, or death with the intention to improve prediction of deterioration due to COVID-19. We compare PICTURE to the Epic Deterioration Index (EDI), a widespread system which has recently been assessed for use to triage COVID-19 patients.Methods The PICTURE model was trained and validated on a cohort of hospitalized non-COVID-19 patients using electronic health record data from 2014-2018. It was then applied to two hold-out test sets: non-COVID-19 patients from 2019 and patients testing positive for COVID-19 in 2020. PICTURE results were aligned to the EDI for head-to-head comparison via Area Under the Receiver Operator Curve (AUROC) and Area Under the Precision Recall Curve (AUPRC). We compared the models’ ability to predict an adverse event (defined as ICU transfer, mechanical ventilation use, or death) at two levels of granularity: (1) maximum score across an encounter with a minimum lead time before the first adverse event and (2) predictions at every observation with instances in the last 24 hours before the adverse event labeled as positive. PICTURE and the EDI were also compared on the encounter level using different lead times extending out to 24 hours. Shapley values were used to provide explanations for PICTURE predictions.Results PICTURE successfully delineated between high- and low-risk patients and consistently outperformed the EDI in both of our cohorts. In non-COVID-19 patients, PICTURE achieved an AUROC (95% CI) of 0.819 (0.805 - 0.834) and AUPRC of 0.109 (0.089 - 0.125) on the observation level, compared to the EDI AUROC of 0.762 (0.746 - 0.780) and AUPRC of 0.077 (0.062 - 0.090). On COVID-19 positive patients, PICTURE achieved an AUROC of 0.828 (0.794 - 0.869) and AUPRC of 0.160 (0.089 - 0.199), while the EDI scored an AUROC of 0.792 (0.754 - 0.835) and AUPRC of 0.131 (0.092 - 0.159). The most important variables influencing PICTURE predictions in the COVID-19 cohort were a rapid respiratory rate, a high level of oxygen support, low oxygen saturation, and impaired mental status (Glasgow coma score).Conclusion The PICTURE model is more accurate in predicting adverse patient outcomes for both general ward patients and COVID-19 positive patients in our cohorts compared to the EDI. The ability to consistently anticipate these events may be especially valuable when considering a potential incipient second wave of COVID-19 infections. PICTURE also has the ability to explain individual predictions to clinicians by ranking the most important features for a prediction. The generalizability of the model will require testing in other health care systems for validation.Competing Interest StatementChristopher E. Gillies, Richard P. Medlin and Kevin R. Ward submitted a patent regarding our machine learning methodologies presented in this paper through the University of Michigan's Office of Technology Transfer.Funding StatementFunded in part by the Michigan Institute for Data Science (MIDAS) "Propelling Original Data Science (PODS) Mini-Grants for COVID-19 Research" award.Author DeclarationsI confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.YesThe details of the IRB/oversight body that provided approval or exemption for the research described are given below:The study protocol was approved by the University of Michigan's Institutional Review Board (HUM00092309).All necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived.YesI understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).YesI have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.YesData and source code are not publicly available at this time. %U https://www.medrxiv.org/content/medrxiv/early/2020/07/10/2020.07.08.20145078.full.pdf