Skip to main content
medRxiv
  • Home
  • About
  • Submit
  • ALERTS / RSS
Advanced Search

Development and Validation of a Survival Calculator for Hospitalized Patients with COVID-19

Todd J. Levy, Safiya Richardson, Kevin Coppa, Douglas P. Barnaby, Thomas McGinn, Lance B. Becker, Karina W. Davidson, Stuart L. Cohen, Jamie S. Hirsch, Theodoros P. Zanos, And the Northwell & Maimonides COVID-19 Research Consortium, Hannah Bodenstein, Shubham Debnath, Andrew J. Dominello, Louise Falzon, Michael Gitman, Jay M. Goldstein, Crystal Herron, Eun-Ji Kim, Lawrence Lau, Zachary S. Lockerman, Alexander Makhnevich, Jazmin N. Mogavero, Ernesto P. Molmenti, Marc d. Paradis, Viktor Tóth
doi: https://doi.org/10.1101/2020.04.22.20075416
Todd J. Levy
aInstitute of Bioelectronic Medicine, Feinstein Institutes for Medical Research, Northwell Health, Manhasset, NY
MS
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Safiya Richardson
bDonald and Barbara Zucker School of Medicine at Hofstra/Northwell, Northwell Health, Hempstead, NY
cInstitute of Health Innovations and Outcomes Research, Feinstein Institutes for Medical Research, Northwell Health, Manhasset, NY
MD, MPH
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Kevin Coppa
dDepartment of Information Services, Northwell Health, New Hyde Park, NY
BS
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Douglas P. Barnaby
bDonald and Barbara Zucker School of Medicine at Hofstra/Northwell, Northwell Health, Hempstead, NY
cInstitute of Health Innovations and Outcomes Research, Feinstein Institutes for Medical Research, Northwell Health, Manhasset, NY
MD, MSc
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Thomas McGinn
bDonald and Barbara Zucker School of Medicine at Hofstra/Northwell, Northwell Health, Hempstead, NY
cInstitute of Health Innovations and Outcomes Research, Feinstein Institutes for Medical Research, Northwell Health, Manhasset, NY
MD, MPH
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Lance B. Becker
aInstitute of Bioelectronic Medicine, Feinstein Institutes for Medical Research, Northwell Health, Manhasset, NY
bDonald and Barbara Zucker School of Medicine at Hofstra/Northwell, Northwell Health, Hempstead, NY
MD
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Karina W. Davidson
bDonald and Barbara Zucker School of Medicine at Hofstra/Northwell, Northwell Health, Hempstead, NY
cInstitute of Health Innovations and Outcomes Research, Feinstein Institutes for Medical Research, Northwell Health, Manhasset, NY
PhD, MASc
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Stuart L. Cohen
bDonald and Barbara Zucker School of Medicine at Hofstra/Northwell, Northwell Health, Hempstead, NY
cInstitute of Health Innovations and Outcomes Research, Feinstein Institutes for Medical Research, Northwell Health, Manhasset, NY
MD
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Jamie S. Hirsch
bDonald and Barbara Zucker School of Medicine at Hofstra/Northwell, Northwell Health, Hempstead, NY
cInstitute of Health Innovations and Outcomes Research, Feinstein Institutes for Medical Research, Northwell Health, Manhasset, NY
dDepartment of Information Services, Northwell Health, New Hyde Park, NY
MD, MA, MSB
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Theodoros P. Zanos
aInstitute of Bioelectronic Medicine, Feinstein Institutes for Medical Research, Northwell Health, Manhasset, NY
bDonald and Barbara Zucker School of Medicine at Hofstra/Northwell, Northwell Health, Hempstead, NY
PhD
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • For correspondence: tzanos{at}northwell.edu
Hannah Bodenstein
eMaimonides Medical Center, Brooklyn, NY
MBA, BSN, RN
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Shubham Debnath
aInstitute of Bioelectronic Medicine, Feinstein Institutes for Medical Research, Northwell Health, Manhasset, NY
PhD
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Andrew J. Dominello
cInstitute of Health Innovations and Outcomes Research, Feinstein Institutes for Medical Research, Northwell Health, Manhasset, NY
BA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Louise Falzon
cInstitute of Health Innovations and Outcomes Research, Feinstein Institutes for Medical Research, Northwell Health, Manhasset, NY
BA, PGDipInf
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Michael Gitman
bDonald and Barbara Zucker School of Medicine at Hofstra/Northwell, Northwell Health, Hempstead, NY
MD
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Jay M. Goldstein
fManagement Information Systems, Maimonides Medical Center, Brooklyn, NY
MBA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Crystal Herron
cInstitute of Health Innovations and Outcomes Research, Feinstein Institutes for Medical Research, Northwell Health, Manhasset, NY
PhD
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Eun-Ji Kim
bDonald and Barbara Zucker School of Medicine at Hofstra/Northwell, Northwell Health, Hempstead, NY
cInstitute of Health Innovations and Outcomes Research, Feinstein Institutes for Medical Research, Northwell Health, Manhasset, NY
MD, MS
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Lawrence Lau
bDonald and Barbara Zucker School of Medicine at Hofstra/Northwell, Northwell Health, Hempstead, NY
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Zachary S. Lockerman
eMaimonides Medical Center, Brooklyn, NY
MD, MBA, FACG, CPE
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Alexander Makhnevich
bDonald and Barbara Zucker School of Medicine at Hofstra/Northwell, Northwell Health, Hempstead, NY
MD
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Jazmin N. Mogavero
cInstitute of Health Innovations and Outcomes Research, Feinstein Institutes for Medical Research, Northwell Health, Manhasset, NY
MA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Ernesto P. Molmenti
bDonald and Barbara Zucker School of Medicine at Hofstra/Northwell, Northwell Health, Hempstead, NY
cInstitute of Health Innovations and Outcomes Research, Feinstein Institutes for Medical Research, Northwell Health, Manhasset, NY
MD, PhD, MBA, FACS
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Marc d. Paradis
gDepartment of Data Strategy & Ventures, Northwell Health, Manhasset, NY
SM
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Viktor Tóth
aInstitute of Bioelectronic Medicine, Feinstein Institutes for Medical Research, Northwell Health, Manhasset, NY
MSc
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • Abstract
  • Full Text
  • Info/History
  • Metrics
  • Data/Code
  • Preview PDF
Loading

Abstract

Background Chinese studies reported predictors of severe disease and mortality associated with coronavirus disease 2019 (COVID-19). A generalizable and simple survival calculator based on data from US patients hospitalized with COVID-19 has not yet been introduced.

Objective Develop and validate a clinical tool to predict 7-day survival in patients hospitalized with COVID-19.

Design Retrospective and prospective cohort study.

Setting Thirteen acute care hospitals in the New York City area.

Participants Adult patients hospitalized with a confirmed diagnosis of COVID-19. The development and internal validation cohort included patients hospitalized between March 1 and May 6, 2020. The external validation cohort included patients hospitalized between March 1 and May 5, 2020.

Measurements Demographic, laboratory, clinical, and outcome data were extracted from the electronic health record. Optimal predictors and performance were identified using least absolute shrinkage and selection operator (LASSO) regression with receiver operating characteristic curves and measurements of area under the curve (AUC).

Results The development and internal validation cohort included 11□095 patients with a median age of 65 years [interquartile range (IQR) 54-77]. Overall 7-day survival was 89%. Serum blood urea nitrogen, age, absolute neutrophil count, red cell distribution width, oxygen saturation, and serum sodium were identified as the 6 optimal of 42 possible predictors of survival. These factors constitute the NOCOS (Northwell COVID-19 Survival) Calculator. Performance in the internal validation, prospective validation, and external validation were marked by AUCs of 0.86, 0.82, and 0.82, respectively.

Limitations All participants were hospitalized within the New York City area.

Conclusions The NOCOS Calculator uses 6 factors routinely available at hospital admission to predict 7-day survival for patients hospitalized with COVID-19. The calculator is publicly available at https://feinstein.northwell.edu/NOCOS.

Trial registration N/A

Funding Source This work was supported by grants R24AG064191 from the National Institute on Aging, R01LM012836 from the National Library of Medicine, and K23HL145114 from the National Heart Lung and Blood Institute.

INTRODUCTION

The World Health Organization designated coronavirus disease 2019 (COVID-19) as a global pandemic on March 11, 2020, with over 1 million confirmed cases worldwide (1). Estimates of severe disease range from 20% to 30%, and fatality rates range from 2% to 7% (2, 3). As healthcare facilities around the world struggle to provide care for rising numbers of critically ill patients, evidence-based tools to assist with prognosis and estimating disease severity are becoming increasingly important (4). These tools can guide conversations with patients and families, advise therapeutic decisions (e.g., admission to the intensive care unit), and align treatment plans with the likelihood of benefit (5).

Some clinical prediction tools have been established to estimate survival in patients with pneumonia or hospitalized with severe illness, including the Sequential Organ Failure Assessment (SOFA) and CURB-65 Scores. However, these tools have not been validated in patients with COVID-19. A recent study reported a clinical risk score for hospitalized patients with this disease (6). While the score relies on data routinely available at admission, it uses unstructured data, cannot predict survival alone, and is based on a cohort of patients hospitalized in China. Several other models predict outcomes in patients with COVID-19. However, many of these models are not peer-reviewed and are at a high risk of bias because of non-representative samples of control patients (7).

Our objective was to develop and externally validate a clinical prediction tool to estimate 7-day survival in patients hospitalized with COVID-19 in the United States. For this tool, we aimed to use exclusively discrete data points from the electronic health record (EHR), forgoing symptom-related records and radiology reads. By including only objective, structured data points that are routinely available at hospital admission, we could reduce ambiguity, improve external performance, and ensure that the tool could be used in most acute-care settings.

METHODS

Study Design

This study includes a retrospective analysis for development, retrospective internal validation, prospective internal validation, and external validation of a model to predict survival of patients hospitalized with COVID-19. The development cohort included patients admitted to 11 of 12 acute care facilities in the Northwell Health system between March 1 and April 23, 2020. The internal retrospective validation cohort included patients admitted to the remaining acute care tertiary facility in the Northwell Health System, Long Island Jewish Medical Center, between March 1 and May 7, 2020. Long Island Jewish Medical Center has the largest number of hospitalized patients with COVID-19 in the Northwell Health system. The internal prospective validation cohort included patients admitted to all 12 acute care facilities in the Northwell Health system between April 24 and May 6, 2020. The external validation cohort included patients admitted to Maimonides Medical Center, an affiliate of the Northwell Health system, between March 1 and May 12, 2020. The final date of follow-up was May 7, 2020 for the internal validation cohorts and May 12, 2020 for the external validation cohort (Figure 1A).

Figure 1.
  • Download figure
  • Open in new tab
Figure 1. Study design and predictive performance of the NOCOS Calculator.

(A) Training and validation datasets used to create and test the predictive performance of the NOCOS Calculator. The datasets include internal retrospective, internal prospective, and external datasets comprising 13□764 patients. Note that the internal retrospective and prospective validation sets overlap by 624 patients admitted to Long Island Jewish Medical Center. (B) Importance of the 6 predictors captured by the coefficients of the NOCOS Calculator. NOCOS = Northwell COVID-19 Survival.

Patients were included if they were adults (≥ 18 years old) admitted to the hospital with COVID-19 confirmed by a positive result from polymerase chain reaction testing of a nasopharyngeal sample. Clinical outcomes (i.e., discharges, mortality, length of stay) were monitored until the final date of follow-up. Patients were excluded if they received invasive mechanical ventilation before inpatient admission, either before presentation to or during their stay in the emergency department. Patients were also excluded if their length of stay was less than 7 days and they were still hospitalized on the final date of follow-up.

This study was approved by the Institutional Review Boards at Northwell Health and Maimonides Medical Center as minimal-risk research that used data collected for routine clinical practice, and as such, waived the requirement for informed consent.

Data Acquisition

Data were collected from the enterprise electronic health record (EHR; Sunrise Clinical Manager, Allscripts, Chicago, IL). Transfers from 1 in-system hospital to another were merged and considered 1 hospital visit. Data collected for the development and internal validation of the tool included patient demographic information, comorbidities, laboratory values, and outcomes (i.e., death, length of stay, discharge). Data collected for the external validation included only the 6 predictor variables found in the development and internal validation process, length of stay, and final outcome (i.e., death, discharge).

Potential Predictive Variables

Potential predictive variables were included if they were available for more than half of study patients at the time of admission. This approach ensured that the results would contain data points routinely available at admission. Continuous variables are presented as median and interquartile range (IQR), and categorical variables are expressed as the number of patients and percentage.

Demographic variables included age, gender, race, ethnicity, and language preference as English or non-English. Vitals signs included systolic blood pressure, diastolic blood pressure, heart rate, respiratory rate, oxygen saturation, temperature, body mass index, height, and weight. Comorbidities included coronary artery disease, diabetes, hypertension, heart failure, lung disease, and kidney disease. Laboratory variables included white blood cell count, absolute neutrophil count, automated lymphocyte count, automated eosinophil count, automated monocyte count, hemoglobin, red cell distribution width, automated platelet count, serum sodium, serum potassium, serum chloride, serum carbon dioxide, serum blood urea nitrogen, serum creatinine, estimated glomerular filtration rate, serum glucose, serum albumin, serum bilirubin, serum alkaline phosphatase, alanine aminotransferase, aspartate aminotransferase, and lactate.

Outcomes

Outcomes collected included death, length of stay, and discharge. The primary outcome was 7-day survival. Patients who were discharged alive on any hospital day before or on hospital day 7 were considered to have survived. Patients who were alive and still in-hospital on hospital day 7 were considered to have survived. Patients who died before or on hospital day 7 were considered to have expired. Patients who were alive and still in-hospital at the study endpoint with a length of stay less than 7 days were excluded from the study.

Prediction Model Development

The model was developed by analyzing 42 potential predictors for the patients hospitalized in 11 of 12 hospitals within the Northwell Health system and discharged on or before April 23, 2020 (Figure 1A). Patients hospitalized at Long Island Jewish Medical Center or discharged after April 23 were used for the internal retrospective and internal prospective validations, respectively. Least Absolute Shrinkage and Selection Operator (LASSO) regression was used to identify predictors that, when linearly combined, predict the survival of hospitalized patients with COVID-19 (8). Missing measurements were imputed using mean imputation. All analyses were performed in MATLAB 2019b (The Mathworks, Inc., Natick, MA).

By including an L1-norm regularization term that promotes sparsity, LASSO regression determines a subset of measurements in which only the strongest predictors remain in the model. The magnitudes of the coefficients relate to the predictive values of the normalized measurements, while coefficients of non-predictive measurements converge exactly to 0. The data are normalized by taking the z-score, which puts all measurements on the same scale. The mean and standard deviation of the measurements with coefficients that are not 0 are stored during training and applied to test data.

The training set was evaluated with the model using 50-fold cross-validation to prevent overfitting. The class-conditional likelihood functions of the LASSO predictions for survival past 7 days and expiration before 7 days were estimated, and the posterior probability of survival past 7 days was evaluated using Bayes Theorem. The regularization factor λ is a hyperparameter that was swept over a range while evaluating the area under the receiver operating characteristic (ROC) curve. After optimizing for λ, the number of predictors was fixed at 6 inputs. The variables identified were used to construct the Northwell COVID-19 Survival (NOCOS) Calculator, available publicly at https://feinstein.northwell.edu/NOCOS.

Prediction Model Validation

The generalizability of the NOCOS Calculator was validated with the retrospective cohort from Long Island Jewish Medical Center, the internal prospective cohort, and the external cohort. The predictive performance of the model was assessed at the time of admission and every 2 days within the hospitalization via ROC and precision recall (PR) curves and the Area Under the Curve (AUC). We also tested the predictive value of the SOFA Score and CURB-65 Score for pneumonia severity, and we compared the AUCs for each score using the nonparametric DeLong method (9, 10).

To determine the performance of survival predictions for all calculators, operating points can be established by choosing thresholds on the probability scores. We chose 3 operating points for each calculator and provided the numbers of true positives, true negatives, false positives, and false negatives, as well as the positive predictive value (PPV) and negative predictive value (NPV) for each calculator.

Calculation of SOFA and CURB-65 Scores

In 2 of the test datasets, data were assessed with the SOFA and CURB-65 Scores. The SOFA Score numerically quantifies the severity of failed organs based on PaO2/FiO2 (11), the Glasgow Coma Scale, mechanical ventilation (yes/no), platelets, bilirubin, mean arterial pressure or administration of vasoactive agents, and creatinine. Because PaO2 was not captured for most patients, we used a formula with SpO2 and FiO2 (Appendix Table 4). For patients missing a Glasgow Coma Scale score, the patient’s mental status was assessed using nursing documentation of mental status and level of consciousness. Missing data on all other variables were limited and treated similar to our imputations. No patients were missing data on ventilation status. The CURB-65 score is another mortality risk score based on confusion, blood urea nitrogen, respiratory rate, blood pressure, and age. For this score, a patient’s confusion level was assessed using nursing documentation.

View this table:
  • View inline
  • View popup
  • Download powerpoint
Appendix Table 1.

Summary of the 6 Predictor Variables from External Validation Data from Patients hospitalized at Maimonides Medical Center

View this table:
  • View inline
  • View popup
  • Download powerpoint
Appendix Table 2.

Confusion Matrices for Multiple Operating Points for the 3 Calculators Tested on the Prospective Dataset.

View this table:
  • View inline
  • View popup
Appendix Table 3.

Confusion Matrices for Multiple Operating Points for the NOCOS Calculator Tested on the External Dataset.

View this table:
  • View inline
  • View popup
  • Download powerpoint
Appendix Table 4.

Determination of PaO2/FiO2 Score Based on SpO2 and FiO2 Values.

Role of the Funding Source

This work was supported by grants R24AG064191 from the National Institute on Aging, R01LM012836 from the National Library of Medicine, and K23HL145114 from the National Heart Lung and Blood Institute. The views expressed in this paper are those of the authors and do not represent the views of the National Institutes of Health, the United States Department of Health and Human Services, or any other government entity.

RESULTS

Patient Characteristics

A total of 11 919 adult patients were hospitalized at the 12 Northwell Health acute care facilities between March 1 and May 7, 2020. Of these patients, 360 (3.02%) were excluded because they were still in the hospital at the study end point with a length of stay less than 7 days; 460 (3.86%) were excluded because they were transferred to a hospital outside of the health system and their outcomes were unknown; and 4 (0.03%) were excluded because they expired but were not marked as discharged in the EHR. The remaining 11□095 (93.09%) patients were included in the development and internal validation cohort. These patients had a median age of 65 years [IQR 54-77], and 42% were female. Overall 7-day survival was 89%. At the study end point, 10□207 (92%) patients were discharged alive or expired and 888 (8%) were still in the hospital. Baseline characteristics of included patients are presented in Table 1.

View this table:
  • View inline
  • View popup
Table 1.

Demographic, Clinical, and Laboratory Data of All Patients Hospitalized at Northwell Health

Survival Prediction Model

Development

To determine the predictors of survival, data were collected from patients hospitalized in 11 of 12 Northwell Health hospitals (n = 6162) (Figure 1A). The optimal predictors of survival, in decreasing order of relative predictive strength, were serum blood urea nitrogen, patient age, absolute neutrophil count, red cell distribution width, oxygen saturation, and serum sodium. For each predictor, the magnitude of the coefficient indicates the relative strength of the predictor in determining the outcome, and the sign of the coefficient corresponds to the sign of the correlation between the predictor and the outcome (Figure 1B).

Validation

For the internal retrospective validation, data were collected from patients hospitalized in Long Island Jewish Medical Center (n = 2229). These data were analyzed with NOCOS, and then the ROC and PR curves and AUC values were determined. The NOCOS Calculator resulted in an AUC of 0.86, which significantly outperformed the SOFA (AUC = 0.70; p < 0.05) and CURB-65 (AUC = 0.81; p < 0.05) Scores (Figure 2A).

Figure 2.
  • Download figure
  • Open in new tab
Figure 2. Predictive performance of the NOCOS Calculator on the internal retrospective, internal prospective, and external datasets.

ROC and PR curves for the (A) internal retrospective validation with patients hospitalized at Long Island Jewish Hospital (n = 2229), (B) prospective validation with patients hospitalized across all 12 Northwell Health hospitals (n = 3328), and (C) external validation with patients hospitalized at Maimonides Medical Center (n = 2669). AUC = area under the curve; FPR = false positive rate; NOCOS = Northwell COVID-19 Survival; PR = precision recall; ROC = receiver operating characteristic; SOFA = Sequential Organ Failure Assessment; TPR = true positive rate.

For the internal prospective validation, data were collected from patients discharged from all 12 Northwell hospitals (n = 3328) (Figure 1A). Based on these data, the NOCOS Calculator (AUC = 0.82) significantly outperformed the SOFA (AUC = 0.64; p < 0.05) and CURB-65 (AUC = 0.72; p < 0.05) Scores (Figure 2B).

For the external validation, data were collected from patients hospitalized at Maimodines Medical Center (n = 2669) (Figure 1A). The NOCOS Calculator yielded a comparable AUC of 0.82 (Figure 2C); however, the SOFA and CURB-65 Scores were not readily available for this dataset because components of each score were not documented for all patients.

Test Characteristics of the NOCOS Calculator, SOFA Score, and CURB-65 Score

To determine the performance of survival predictions for all calculators, operating points can be established by choosing thresholds on the probability scores. We chose 3 different operating points for each calculator for the internal retrospective validation (Long Island Jewish Medical Center) data and provided the number of true positives, true negatives, false positives, and false negatives. We also provided the PPV and NPV for each case (Table 2). In all cases, the NOCOS Calculator outperformed the SOFA and CURB-65 Scores. See Appendix Table 2 for metrics at the operating points of data from the internal retrospective validation (Long Island Jewish Medical Center), and see Appendix Table 3 for metrics from the external validation (Maimonides Medical Center).

View this table:
  • View inline
  • View popup
Table 2.

Confusion Matrices for Multiple Operating Points for the 3 Calculators Tested on Data from the Long Island Jewish Medical Center Dataset.

The NOCOS Calculator can also be reevaluated with updated labs and vitals as the patient’s condition progresses. The performance of the NOCOS calculator remained relatively stable over 10 days (Figure 3).

Figure 3.
  • Download figure
  • Open in new tab
Figure 3. Stability of the predictive performance of the NOCOS Calculator across hospitalization days.

AUC capturing the predictive performance of the NOCOS Calculator. The performance remained stable when the most recent values of the predictors were updated for up to 10 days during hospitalization after admission from the emergency department. AUC = Area Under the Curve; NOCOS = Northwell COVID-19 Survival.

DISCUSSION

This study is the first to develop a model, the NOCOS Calculator, that predicts survival of patients hospitalized with COVID-19 in the United States. We created and validated the NOCOS Calculator with data on almost 14 000 patients, using only 6 clinical data points typically available to clinicians within the first 60 minutes of patient presentation. All data points are available as discrete inputs in most commercial EHRs, supporting that this calculator could be readily incorporated into tools to support clinical decisions. This calculator is publicly available at https://feinstein.northwell.edu/NOCOS and can be used by clinicians to estimate the probability of survival for their patients.

Several elements of the NOCOS Calculator have been either established as prognostic markers in other populations or identified as risk factors for severe illness or death in patients with COVID-19. Older age and elevated blood urea nitrogen (a marker of kidney dysfunction) have both been associated with increased mortality risk in patients with COVID-19.(12, 13) Hypoxemia, measured by lower levels of blood oxygen, has also been linked to increased mortality in this population (14). Neutrophil count, either individually or paired in a ratio with lymphocytes, also predicts disease severity in COVID-19 patients (15). While serum sodium has not yet been linked to COVID-19, it (16) has been independently and consistently associated with negative outcomes in other populations (17) and disease states.(18, 19) Elevated values of red cell distribution width often suggest chronic disease states and inflammation.(20, 21) An increased red cell distribution width may also be an effect of COVID-19 on iron displacement of the heme molecule, leading to impaired red blood cells, free-radical formation, and a toxic effect to the lungs.(22)

The NOCOS Calculator performs well with the 6 early measurements, and it retains its predictive performance as these measurements are updated over at least 10 days throughout the hospitalization of the patient (Figure 3). This finding supports that the most up-to-date values of the 6 measures can accurately predict survival. Moreover, while we present the calculator output as a probability score, a specific operating point can be chosen to provide a binary outcome prediction with significant accuracy. Stakeholders can choose an operating point, and local clinical teams can adjust thresholds toward a more stringent or risk-averse solution (Table 2) based on the rapidly changing needs during this pandemic.

Due to the challenges that arise during the ongoing COVID-19 pandemic, we need robust tools to aid in making complex clinical decisions. Using well-known clinical calculators, such as the SOFA or CURB-65 Scores, can be useful. However, these scores are limited by their accuracy and the ease of collecting necessary measurements to construct the scores. They also use input variables, such as confusion (for CURB-65) and the Glasgow Coma Scale (for SOFA), both of which are ambiguous, difficult to measure, and frequently unavailable, as shown by our external validation dataset. We found that the NOCOS Calculator consistently outperformed both the CURB-65 and SOFA scores in both our validation datasets.

Limitations

The study population only included patients within the New York City metropolitan area. However, given the diverse demographic population of the region, we expect the model to generalize to patients at centers outside of this geographic area. The data were collected entirely from EHR reports, which supported robust and rapid analysis of a large cohort of patients. However, we did not include data elements that would require manual chart review. Due to the retrospective study design, not all laboratory tests were completed on all patients, and the performance of these variables could not be adequately assessed. To optimize for usability and portability, the analysis was designed to be linear and to include a minimum number of predictors. Non-linear or convolutional/recurrent models may provide improved performance but might not be easily used at all centers.

Conclusion

This study is the first to develop and externally validate a simple predictive model of survival for hospitalized patients with COVID-19 based on structured, objective data that is routinely available at admission in the United States. Serum blood urea nitrogen, age, absolute neutrophil count, red cell distribution width, oxygen saturation, and serum sodium were identified as the 6 optimal predictors of survival. The NOCOS Calculator can predict survival more accurately than commonly used survival predictors.

Data Availability

The data that support the findings of this study are available on request from COVID19{at}northwell.edu. The data are not publicly available due to restrictions as it could compromise the privacy of research participants.

http://feinstein.northwell.edu/nocos

Acknowledgments

We acknowledge and honor all our Northwell team members who consistently put themselves in harm’s way during the COVID-19 pandemic. We dedicate this article to them, as their vital contribution to knowledge about COVID-19 and sacrifices on the behalf of patients made it possible.

References

  1. 1.↵
    The Center for Systems Science and Engineering (CSSE) at Johns Hopkins University [Internet], Baltimore, MD: Center for Systems Science and Engineering; 2020 [cited 2020 May 29], Available from: https://gisanddata.maps.arcgis.com/apps/opsdashboard/index.html#/bda7594740fd40299423467b48e9ecf6.
  2. 2.↵
    Livingston E, Bucher K. Coronavirus Disease 2019 (COVID-19) in Italy. JAMA. Epub 2020 Mar 17.
  3. 3.↵
    Wu Z, McGoogan JM. Characteristics of and Important Lessons From the Coronavirus Disease 2019 (COVID-19) Outbreak in China: Summary of a Report of 72314 Cases From the Chinese Center for Disease Control and Prevention. JAMA. Epub 2020 Feb 24.
  4. 4.↵
    Myers J, Cosby R, Gzik D, et al. Provider Tools for Advance Care Planning and Goals of Care Discussions: A Systematic Review. Am J Hosp Palliat Care. 2018;35(8):1123–32.
    OpenUrlCrossRef
  5. 5.↵
    Brajer N, Cozzi B, Gao M, et al. Prospective and External Evaluation of a Machine Learning Model to Predict In-Hospital Mortality of Adults at Time of Admission. JAMA Netw Open. 2020;3(2):e1920733.
    OpenUrl
  6. 6.↵
    Liang W, Liang H, Ou L, et al. Development and Validation of a Clinical Risk Score to Predict the Occurrence of Critical Illness in Hospitalized Patients With COVID-19. JAMA Intern Med. 2020;12: e202033.
    OpenUrl
  7. 7.↵
    Wynants L, Van Calster B, Bonten MM, et al. Prediction models for diagnosis and prognosis of covid-19 infection: systematic review and critical appraisal. BMJ. 2020;369:m1328.
    OpenUrlAbstract/FREE Full Text
  8. 8.↵
    Tibshirani R. Regression Shrinkage and Selection Via the Lasso. J R Stat Soc Series B Methodol. 1996;58(1):267–88.
    OpenUrl
  9. 9.↵
    DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics. 1988;44(3):837–45.
    OpenUrlCrossRefPubMedWeb of Science
  10. 10.↵
    Sun X, Xu W. Fast Implementation of DeLong’s Algorithm for Comparing the Areas Under Correlated Receiver Operating Characteristic Curves. IEEE Signal Process Lett. 2014;21(11):1389–93.
    OpenUrlCrossRef
  11. 11.↵
    Jones AE, Trzeciak S, Kline JA. The Sequential Organ Failure Assessment score for predicting outcome in patients with severe sepsis and evidence of hypoperfusion at the time of emergency department presentation. Crit Care Med. 2009;37(5):1649–54.
    OpenUrlCrossRefPubMedWeb of Science
  12. 12.↵
    Zhou F, Yu T, Du R, et al. Clinical course and risk factors for mortality of adult inpatients with COVID-19 in Wuhan, China: a retrospective cohort study. Lancet. 2020;395(10229):1054–62.
    OpenUrlCrossRefPubMed
  13. 13.↵
    Cheng Y, Luo R, Wang K, et al. Kidney disease is associated with in-hospital death of patients with COVID-19. Kidney Int. 2020;97(5):829–38.
    OpenUrlCrossRefPubMed
  14. 14.↵
    Xie J, Covassin N, Fan Z, et al. Association Between Hypoxemia and Mortality in Patients With COVID-19. Mayo Clin Proc. Epub 2020 Apr 11.
  15. 15.↵
    Shang W, Dong J, Ren Y, et al. The value of clinical parameters in predicting the severity of COVID-19. J Med Virol. Epub 2020 May 22.
  16. 16.↵
    Steer J, Gibson J, Bourke SC. The DECAF Score: predicting hospital mortality in exacerbations of chronic obstructive pulmonary disease. Thorax. 2012;67(11):970–6.
    OpenUrlAbstract/FREE Full Text
  17. 17.↵
    Zhang X, Li XY. Prevalence of hyponatremia among older inpatients in a general hospital. Eur Geriatr Med. Epub 2020 May 7.
  18. 18.↵
    Kim EJ, Jeong MH, Kim JH, et al. Clinical impact of admission hyperglycemia on in-hospital mortality in acute myocardial infarction patients. Int J Cardiol. 2017;236:9–15.
    OpenUrlCrossRef
  19. 19.↵
    Ruf AE, Kremers WK, Chavez LL, et al. Addition of serum sodium into the MELD score predicts waiting list mortality better than MELD alone. Liver Transpl. 2005;11(3):336–43.
    OpenUrlCrossRefPubMedWeb of Science
  20. 20.↵
    Salvagno GL, Sanchis-Gomar F, Picanza A, et al. Red blood cell distribution width: A simple parameter with multiple clinical applications. Crit Rev Clin Lab Sci. 2015;52(2):86–105.
    OpenUrlCrossRefPubMed
  21. 21.↵
    Wang B, Gong Y, Ying B, et al. Relation between Red Cell Distribution Width and Mortality in Critically Ill Patients with Acute Respiratory Distress Syndrome. Biomed Res Int. 2019;2019:1942078.
    OpenUrl
  22. 22.↵
    Liu W, Li H. COVID-19: Attacks the 1-Beta Chain of Hemoglobin and Captures the Porphyrin to Inhibit Human Heme Metabolism. ChemRxiv [Preprint] 2020 [cited 2020 May 29]. Available from: https://doi.org/10.26434/chemrxiv.11938173.v5
Back to top
PreviousNext
Posted June 02, 2020.
Download PDF
Data/Code
Email

Thank you for your interest in spreading the word about medRxiv.

NOTE: Your email address is requested solely to identify you as the sender of this article.

Enter multiple addresses on separate lines or separate them with commas.
Development and Validation of a Survival Calculator for Hospitalized Patients with COVID-19
(Your Name) has forwarded a page to you from medRxiv
(Your Name) thought you would like to see this page from the medRxiv website.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.
Share
Development and Validation of a Survival Calculator for Hospitalized Patients with COVID-19
Todd J. Levy, Safiya Richardson, Kevin Coppa, Douglas P. Barnaby, Thomas McGinn, Lance B. Becker, Karina W. Davidson, Stuart L. Cohen, Jamie S. Hirsch, Theodoros P. Zanos, And the Northwell & Maimonides COVID-19 Research Consortium, Hannah Bodenstein, Shubham Debnath, Andrew J. Dominello, Louise Falzon, Michael Gitman, Jay M. Goldstein, Crystal Herron, Eun-Ji Kim, Lawrence Lau, Zachary S. Lockerman, Alexander Makhnevich, Jazmin N. Mogavero, Ernesto P. Molmenti, Marc d. Paradis, Viktor Tóth
medRxiv 2020.04.22.20075416; doi: https://doi.org/10.1101/2020.04.22.20075416
Twitter logo Facebook logo LinkedIn logo Mendeley logo
Citation Tools
Development and Validation of a Survival Calculator for Hospitalized Patients with COVID-19
Todd J. Levy, Safiya Richardson, Kevin Coppa, Douglas P. Barnaby, Thomas McGinn, Lance B. Becker, Karina W. Davidson, Stuart L. Cohen, Jamie S. Hirsch, Theodoros P. Zanos, And the Northwell & Maimonides COVID-19 Research Consortium, Hannah Bodenstein, Shubham Debnath, Andrew J. Dominello, Louise Falzon, Michael Gitman, Jay M. Goldstein, Crystal Herron, Eun-Ji Kim, Lawrence Lau, Zachary S. Lockerman, Alexander Makhnevich, Jazmin N. Mogavero, Ernesto P. Molmenti, Marc d. Paradis, Viktor Tóth
medRxiv 2020.04.22.20075416; doi: https://doi.org/10.1101/2020.04.22.20075416

Citation Manager Formats

  • BibTeX
  • Bookends
  • EasyBib
  • EndNote (tagged)
  • EndNote 8 (xml)
  • Medlars
  • Mendeley
  • Papers
  • RefWorks Tagged
  • Ref Manager
  • RIS
  • Zotero
  • Tweet Widget
  • Facebook Like
  • Google Plus One

Subject Area

  • Health Informatics
Subject Areas
All Articles
  • Addiction Medicine (430)
  • Allergy and Immunology (755)
  • Anesthesia (221)
  • Cardiovascular Medicine (3288)
  • Dentistry and Oral Medicine (364)
  • Dermatology (277)
  • Emergency Medicine (479)
  • Endocrinology (including Diabetes Mellitus and Metabolic Disease) (1169)
  • Epidemiology (13359)
  • Forensic Medicine (19)
  • Gastroenterology (898)
  • Genetic and Genomic Medicine (5147)
  • Geriatric Medicine (481)
  • Health Economics (782)
  • Health Informatics (3264)
  • Health Policy (1140)
  • Health Systems and Quality Improvement (1190)
  • Hematology (429)
  • HIV/AIDS (1017)
  • Infectious Diseases (except HIV/AIDS) (14622)
  • Intensive Care and Critical Care Medicine (912)
  • Medical Education (476)
  • Medical Ethics (127)
  • Nephrology (522)
  • Neurology (4919)
  • Nursing (262)
  • Nutrition (727)
  • Obstetrics and Gynecology (882)
  • Occupational and Environmental Health (795)
  • Oncology (2519)
  • Ophthalmology (723)
  • Orthopedics (280)
  • Otolaryngology (347)
  • Pain Medicine (323)
  • Palliative Medicine (90)
  • Pathology (543)
  • Pediatrics (1299)
  • Pharmacology and Therapeutics (550)
  • Primary Care Research (556)
  • Psychiatry and Clinical Psychology (4205)
  • Public and Global Health (7499)
  • Radiology and Imaging (1704)
  • Rehabilitation Medicine and Physical Therapy (1011)
  • Respiratory Medicine (980)
  • Rheumatology (479)
  • Sexual and Reproductive Health (497)
  • Sports Medicine (424)
  • Surgery (547)
  • Toxicology (72)
  • Transplantation (235)
  • Urology (205)