Skip to main content
medRxiv
  • Home
  • About
  • Submit
  • ALERTS / RSS
Advanced Search

Validation and comparison of PICTURE analytic and Epic Deterioration Index for COVID-19

Brandon C. Cummings, Sardar Ansari, Jonathan R. Motyka, Guan Wang, Richard P. Medlin Jr., Steven L. Kronick, Karandeep Singh, Pauline K. Park, Lena M. Napolitano, Robert P. Dickson, Michael R. Mathis, Michael W. Sjoding, Andrew J. Admon, Kevin R. Ward, Christopher E. Gillies
doi: https://doi.org/10.1101/2020.07.08.20145078
Brandon C. Cummings
1Department of Emergency Medicine, University of Michigan, Ann Arbor
9Michigan Center for Integrative Research in Critical Care (MCIRCC) University of Michigan, Ann Arbor
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • For correspondence: cummingb@med.umich.edu cgillies@med.umich.edu
Sardar Ansari
1Department of Emergency Medicine, University of Michigan, Ann Arbor
9Michigan Center for Integrative Research in Critical Care (MCIRCC) University of Michigan, Ann Arbor
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Jonathan R. Motyka
1Department of Emergency Medicine, University of Michigan, Ann Arbor
9Michigan Center for Integrative Research in Critical Care (MCIRCC) University of Michigan, Ann Arbor
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Guan Wang
1Department of Emergency Medicine, University of Michigan, Ann Arbor
9Michigan Center for Integrative Research in Critical Care (MCIRCC) University of Michigan, Ann Arbor
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Richard P. Medlin Jr.
1Department of Emergency Medicine, University of Michigan, Ann Arbor
9Michigan Center for Integrative Research in Critical Care (MCIRCC) University of Michigan, Ann Arbor
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Steven L. Kronick
1Department of Emergency Medicine, University of Michigan, Ann Arbor
9Michigan Center for Integrative Research in Critical Care (MCIRCC) University of Michigan, Ann Arbor
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Karandeep Singh
2Department of Internal Medicine, University of Michigan, Ann Arbor
3Department of Learning Health Sciences, University of Michigan, Ann Arbor
8Michigan Institute for Data Science (MIDAS), University of Michigan, Ann Arbor
9Michigan Center for Integrative Research in Critical Care (MCIRCC) University of Michigan, Ann Arbor
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Pauline K. Park
4Department of Surgery, University of Michigan, Ann Arbor
9Michigan Center for Integrative Research in Critical Care (MCIRCC) University of Michigan, Ann Arbor
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Lena M. Napolitano
4Department of Surgery, University of Michigan, Ann Arbor
9Michigan Center for Integrative Research in Critical Care (MCIRCC) University of Michigan, Ann Arbor
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Robert P. Dickson
2Department of Internal Medicine, University of Michigan, Ann Arbor
5Department of Microbiology & Immunology, University of Michigan, Ann Arbor
9Michigan Center for Integrative Research in Critical Care (MCIRCC) University of Michigan, Ann Arbor
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Michael R. Mathis
6Department of Anesthesiology, University of Michigan, Ann Arbor
9Michigan Center for Integrative Research in Critical Care (MCIRCC) University of Michigan, Ann Arbor
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Michael W. Sjoding
2Department of Internal Medicine, University of Michigan, Ann Arbor
9Michigan Center for Integrative Research in Critical Care (MCIRCC) University of Michigan, Ann Arbor
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Andrew J. Admon
2Department of Internal Medicine, University of Michigan, Ann Arbor
8Michigan Institute for Data Science (MIDAS), University of Michigan, Ann Arbor
9Michigan Center for Integrative Research in Critical Care (MCIRCC) University of Michigan, Ann Arbor
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Kevin R. Ward
1Department of Emergency Medicine, University of Michigan, Ann Arbor
7Department of Biomedical Engineering, University of Michigan, Ann Arbor
8Michigan Institute for Data Science (MIDAS), University of Michigan, Ann Arbor
9Michigan Center for Integrative Research in Critical Care (MCIRCC) University of Michigan, Ann Arbor
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Christopher E. Gillies
1Department of Emergency Medicine, University of Michigan, Ann Arbor
8Michigan Institute for Data Science (MIDAS), University of Michigan, Ann Arbor
9Michigan Center for Integrative Research in Critical Care (MCIRCC) University of Michigan, Ann Arbor
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • For correspondence: cummingb@med.umich.edu cgillies@med.umich.edu
  • Abstract
  • Full Text
  • Info/History
  • Metrics
  • Data/Code
  • Preview PDF
Loading

Abstract

Introduction The 2019 coronavirus (COVID-19) has led to unprecedented strain on healthcare facilities across the United States. Accurately identifying patients at an increased risk of deterioration may help hospitals manage their resources while improving the quality of patient care. Here we present the results of an analytical model, PICTURE (Predicting Intensive Care Transfers and Other UnfoReseen Events), to identify patients at a high risk for imminent intensive care unit (ICU) transfer, respiratory failure, or death with the intention to improve prediction of deterioration due to COVID-19. We compare PICTURE to the Epic Deterioration Index (EDI), a widespread system which has recently been assessed for use to triage COVID-19 patients.

Methods The PICTURE model was trained and validated on a cohort of hospitalized non-COVID-19 patients using electronic health record data from 2014-2018. It was then applied to two hold-out test sets: non-COVID-19 patients from 2019 and patients testing positive for COVID-19 in 2020. PICTURE results were aligned to the EDI for head-to-head comparison via Area Under the Receiver Operator Curve (AUROC) and Area Under the Precision Recall Curve (AUPRC). We compared the models’ ability to predict an adverse event (defined as ICU transfer, mechanical ventilation use, or death) at two levels of granularity: (1) maximum score across an encounter with a minimum lead time before the first adverse event and (2) predictions at every observation with instances in the last 24 hours before the adverse event labeled as positive. PICTURE and the EDI were also compared on the encounter level using different lead times extending out to 24 hours. Shapley values were used to provide explanations for PICTURE predictions.

Results PICTURE successfully delineated between high- and low-risk patients and consistently outperformed the EDI in both of our cohorts. In non-COVID-19 patients, PICTURE achieved an AUROC (95% CI) of 0.819 (0.805 - 0.834) and AUPRC of 0.109 (0.089 - 0.125) on the observation level, compared to the EDI AUROC of 0.762 (0.746 - 0.780) and AUPRC of 0.077 (0.062 - 0.090). On COVID-19 positive patients, PICTURE achieved an AUROC of 0.828 (0.794 - 0.869) and AUPRC of 0.160 (0.089 - 0.199), while the EDI scored an AUROC of 0.792 (0.754 - 0.835) and AUPRC of 0.131 (0.092 - 0.159). The most important variables influencing PICTURE predictions in the COVID-19 cohort were a rapid respiratory rate, a high level of oxygen support, low oxygen saturation, and impaired mental status (Glasgow coma score).

Conclusion The PICTURE model is more accurate in predicting adverse patient outcomes for both general ward patients and COVID-19 positive patients in our cohorts compared to the EDI. The ability to consistently anticipate these events may be especially valuable when considering a potential incipient second wave of COVID-19 infections. PICTURE also has the ability to explain individual predictions to clinicians by ranking the most important features for a prediction. The generalizability of the model will require testing in other health care systems for validation.

Competing Interest Statement

Christopher E. Gillies, Richard P. Medlin and Kevin R. Ward submitted a patent regarding our machine learning methodologies presented in this paper through the University of Michigan's Office of Technology Transfer.

Funding Statement

Funded in part by the Michigan Institute for Data Science (MIDAS) "Propelling Original Data Science (PODS) Mini-Grants for COVID-19 Research" award.

Author Declarations

I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.

Yes

The details of the IRB/oversight body that provided approval or exemption for the research described are given below:

The study protocol was approved by the University of Michigan's Institutional Review Board (HUM00092309).

All necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived.

Yes

I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).

Yes

I have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.

Yes

Data Availability

Data and source code are not publicly available at this time.

Copyright 
The copyright holder for this preprint is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. All rights reserved. No reuse allowed without permission.
Back to top
PreviousNext
Posted July 10, 2020.
Download PDF
Data/Code
Email

Thank you for your interest in spreading the word about medRxiv.

NOTE: Your email address is requested solely to identify you as the sender of this article.

Enter multiple addresses on separate lines or separate them with commas.
Validation and comparison of PICTURE analytic and Epic Deterioration Index for COVID-19
(Your Name) has forwarded a page to you from medRxiv
(Your Name) thought you would like to see this page from the medRxiv website.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.
Share
Validation and comparison of PICTURE analytic and Epic Deterioration Index for COVID-19
Brandon C. Cummings, Sardar Ansari, Jonathan R. Motyka, Guan Wang, Richard P. Medlin Jr., Steven L. Kronick, Karandeep Singh, Pauline K. Park, Lena M. Napolitano, Robert P. Dickson, Michael R. Mathis, Michael W. Sjoding, Andrew J. Admon, Kevin R. Ward, Christopher E. Gillies
medRxiv 2020.07.08.20145078; doi: https://doi.org/10.1101/2020.07.08.20145078
Reddit logo Twitter logo Facebook logo LinkedIn logo Mendeley logo
Citation Tools
Validation and comparison of PICTURE analytic and Epic Deterioration Index for COVID-19
Brandon C. Cummings, Sardar Ansari, Jonathan R. Motyka, Guan Wang, Richard P. Medlin Jr., Steven L. Kronick, Karandeep Singh, Pauline K. Park, Lena M. Napolitano, Robert P. Dickson, Michael R. Mathis, Michael W. Sjoding, Andrew J. Admon, Kevin R. Ward, Christopher E. Gillies
medRxiv 2020.07.08.20145078; doi: https://doi.org/10.1101/2020.07.08.20145078

Citation Manager Formats

  • BibTeX
  • Bookends
  • EasyBib
  • EndNote (tagged)
  • EndNote 8 (xml)
  • Medlars
  • Mendeley
  • Papers
  • RefWorks Tagged
  • Ref Manager
  • RIS
  • Zotero
  • Tweet Widget
  • Facebook Like
  • Google Plus One

Subject Area

  • Health Informatics
Subject Areas
All Articles
  • Addiction Medicine (228)
  • Allergy and Immunology (504)
  • Anesthesia (110)
  • Cardiovascular Medicine (1240)
  • Dentistry and Oral Medicine (206)
  • Dermatology (147)
  • Emergency Medicine (282)
  • Endocrinology (including Diabetes Mellitus and Metabolic Disease) (532)
  • Epidemiology (10023)
  • Forensic Medicine (5)
  • Gastroenterology (499)
  • Genetic and Genomic Medicine (2456)
  • Geriatric Medicine (238)
  • Health Economics (479)
  • Health Informatics (1644)
  • Health Policy (753)
  • Health Systems and Quality Improvement (636)
  • Hematology (249)
  • HIV/AIDS (533)
  • Infectious Diseases (except HIV/AIDS) (11865)
  • Intensive Care and Critical Care Medicine (626)
  • Medical Education (252)
  • Medical Ethics (75)
  • Nephrology (268)
  • Neurology (2281)
  • Nursing (139)
  • Nutrition (352)
  • Obstetrics and Gynecology (454)
  • Occupational and Environmental Health (537)
  • Oncology (1245)
  • Ophthalmology (377)
  • Orthopedics (134)
  • Otolaryngology (226)
  • Pain Medicine (158)
  • Palliative Medicine (50)
  • Pathology (324)
  • Pediatrics (730)
  • Pharmacology and Therapeutics (313)
  • Primary Care Research (282)
  • Psychiatry and Clinical Psychology (2281)
  • Public and Global Health (4834)
  • Radiology and Imaging (837)
  • Rehabilitation Medicine and Physical Therapy (492)
  • Respiratory Medicine (651)
  • Rheumatology (286)
  • Sexual and Reproductive Health (238)
  • Sports Medicine (227)
  • Surgery (267)
  • Toxicology (44)
  • Transplantation (125)
  • Urology (99)