ABSTRACT
Background and aim The COVID-19 pandemic is putting extraordinary pressure on emergency departments (EDs). To support decision making about hospital admission, we aimed to develop a simple and valid model for predicting mortality and need for admission to an intensive care unit (ICU) in suspected-COVID-19 patients presenting at the ED.
Methods For model development, we included patients that presented at the ED and were admitted to 4 large Dutch hospitals with suspected COVID-19 between March and August 2020, the first wave of the pandemic in the Netherlands. Based on prior literature we included patient characteristics, vital parameters and blood test values, all measured at ED admission, as potential predictors. Logistic regression analyses with post-hoc uniform shrinkage was used to obtain predicted probabilities of in-hospital death and of being admitted to the ICU, both within 28 days after admission. Model performance (AUC; calibration plots, intercepts and slopes) was assessed with temporal validation in patients who presented between September and December 2020 (second wave). We used multiple imputation to account for missing predictor values.
Results The development data included 5,831 patients who presented at the ED and were hospitalized, of whom 629 (10.8%) died and 5,070 (86.9%) were discharged within 28 days after admission. A simple model – named COVID Outcome Prediction in the Emergency Department (COPE) – with linear age and logarithmic transforms of respiratory rate, CRP, LDH, albumin and urea captured most of the ability to predict death within 28 days. Patients who were admitted in the first month of the pandemic had substantially increased risk of death (odds ratio 1.99; 95% CI 1.61-2.47). COPE was well-calibrated and showed good discrimination for predicting death in 3,252 patients of the second wave (AUC in 4 hospitals: 0.82; 0.82; 0.79; 0.83). COPE was also able to identify patients at high risk of needing IC in second wave patients below the age of 70 (AUC 0.84; 0.81), but overestimated ICU admission for low-risk patients. The models are implemented as a web-based application.
Conclusion COPE is a simple tool that is well able to predict mortality and ICU admission for patients who present to the ED with suspected COVID-19 and may help to inform patients and doctors when deciding on hospital admission.
BACKGROUND
The COVID-19 pandemic is putting extraordinary pressure on emergency departments (EDs). Clinical prediction models have the potential to support decision making about hospital admission, but currently available models were assessed to contain a high risk of bias (1, 2). Most important reasons were non-representative selection of control patients, exclusion of patients who had not experienced the event of interest by the end of the study, high risk of model overfitting, and vague reporting. The description of the study population or intended use of the models was often missing, and calibration of the model predictions was rarely assessed.
The recently proposed 4C Mortality Score is a positive exception, but was derived in a selected population of patients admitted to UK hospitals who were seriously ill (mortality rate of 32.2%) and included as predictors the number of comorbidities and the Glasgow Coma Scale, items that were not available in all Dutch databases, because they are not routinely measured for patients with suspected COVID-19 (3, 4).
We aimed to develop and validate a simple and valid model for predicting mortality and the need for ICU in all patients who are suspected to have COVID-19 when presenting at the ED. To better enable implementation in clinical practice, we only included routinely measured patient characteristics, vital parameters and blood test values.
METHODS
Population
19 large Dutch hospital were requested to supply data on COVID-19 patients who were admitted to their hospital. Of those hospitals, Catharina Hospital Eindhoven, Zuyderland Medical Center Heerlen, Isala Hospital Zwolle, Erasmus University Medical Center Rotterdam and Antonius Hospital Sneek actually supplied these data. The data from Antonius Hospital Sneek were not used in the analyses, because of large proportions of missing predictor values.
For model development, we included patients that presented at the ED and were admitted to the hospital with suspected COVID-19 in the first wave of the pandemic, that is until August 2020. Patients being transferred from or to other hospitals were excluded since information on predictors or outcomes was missing. For model validation we included patients that presented at the ED and were admitted to the hospital with suspected COVID-19 in the second wave of the pandemic, that is since September 2020. We used multiple imputation to account for missing predictor values.
Outcomes
The outcomes of interest were in-hospital death and admission to ICU within 28 days after hospital admission. Transfer to a hospice was counted as death.
Predictors
Based on prior literature we included patient characteristics (sex, age, BMI), vital parameters (oxygen saturation, systolic blood pressure, heart rate, respiratory rate [RR], body temperature) and blood test values (C-reactive protein [CRP], lactic dehydrogenase [LDH], D-Dimer, leucocytes, lymphocytes, monocytes, neutrophils, eosinophils, MCV, albumin, bicarbonate, creatinine, sodium, urea), all measured at ED admission, as potential predictors (1). Further we included month of admission to capture potential changes in outcomes over time.
Model development
Logistic regression was used to analyze associations between predictors and outcomes. Based on a full model with restricted cubic splines of all potential predictors, we decided on including non-linear transformations. We selected the most promising predictors into a parsimonious model for easy use in clinical practice. To prevent overfitting, we used bootstrap validation, including backward selection according to our modeling strategy, to estimate a uniform shrinkage factor (5). The regression coefficients of the final model were multiplied by this shrinkage factor, and the model intercept was adjusted to ensure overall calibration of the model.
Model validation
Model performance was assessed with temporal validation in second wave patients, in each of the 4 separate hospitals. We assessed discriminative ability with the area under the operator receiver characteristic curve (AUC) and calibration with calibration plots, calibration intercepts, and calibration slopes. The model-based concordance (mbc) was used to understand the impact of potential differences in case-mix heterogeneity between first wave and second wave patients (6).
We used R-packages mice and rms for imputation of missing values and for regression analyses, respectively (7-9).
RESULTS
Population and outcomes
The development data included 5,831 patients who presented at the ED up until August 2020 and who were admitted to hospital with a suspicion of COVID-19, of whom 629 (10.8%) died and 5,070 (86.9%) were discharged within 28 days after hospital admission. As expected, patients who died – in comparison with patients who were discharged – tended to be more male, at older age, with worse vital parameters (higher RR and HR; lower oxygen saturation), higher blood levels of CRP, LDH, creatinine and urea and lower blood levels of lymphocytes and albumin (Table 1). Similar pattern were visible for 3.252 patients who were admitted to hospital in the second wave of the pandemic, of whom 326 (10.0%) died and 2,854 (87.8%) were discharged within 28 days after admission. Admission to ICU was fully recorded for 2,633 patients in 2 hospitals (214 ICU admissions within 28 days [8.1%]) in the first wave and in 1,466 patients (86 ICU admissions within 28 days [5.9%]) in the second wave of the pandemic.
Prediction of death
Patients who were admitted in the first month of the pandemic in the Netherlands, that is in March 2020, had substantially increased risk of death (Table 2: odds ratio 1.99; 95% confidence interval 1.61-2.47). All models included this correction factor for the first month, to avoid overestimation of risk after the first month of the pandemic. Consequently, to avoid overestimation of the discriminative ability, we limited validation of models in the development data to patients who were admitted from April 2020 onward.
D-dimer was not analyzed in the regression analysis, because 64% and 76% were missing in the development and validation data, respectively (Table 1). Based on a full model with restricted cubic splines of all potential variables, we decided to transform all biomarkers and RR with the natural logarithm, while keeping all other predictor effects linear. The predictive ability of the resulting full multivariable regression model was mainly driven by age, LDH, urea, RR, CRP, Albumin, oxygen saturation and bicarbonate (ORs and Wald statistics in Table 2). A simple model – named COVID Outcome Prediction in the Emergency Department (COPE) – with linear age and logarithmic transforms of RR, CRP, LDH, albumin and urea captured most of the ability to predict death within 28 days (Table 2). Based on internal bootstrap validation we applied a shrinkage factor of 0.93 to the regression coefficients.
COPE was well-calibrated and showed good discrimination for predicting death in 4,498 patients who were admitted from April 2020 in the first wave (Supplementary Figure 1; AUC in 4 hospitals 0.85; 0.86, 0.81, 0.85) and, more importantly, in 3,235 patients in the second wave from September 2020 (Figure 2; AUC in 4 hospitals: 0.82; 0.82; 0.79; 0.83). The decrease in AUC was partly driven by less case mix heterogeneity – expressed by a lower mbc – of second wave patients (Figure 1; mbc in 4 hospitals: 0.81; 0.82; 0.81; 0.82) as compare to first wave patients (Supplementary Figure 1; mbc in 4 hospitals 0.82; 0.85, 0.83, 0.84).
Prediction of ICU admission
The probability of being admitted to the ICU was decreasing with age after the age of 70, probably reflecting the decision not to admit older patients to the ICU. When adjusting for this decreasing age effect after the age of 70 – by including a linear spline with a knot at age 70 the regression model (Supplementary figure 2) – the strongest predictors of death were also predictive of ICU admission within 28 days, but associations were generally weaker (Table 3 vs Table 2). In patients below the age of 70, admitted from April to August 2020, a model with the linear predictor of death calibrated to ICU admission had similar discriminative ability to a model that refitted all the predictor effects (AUC 0.71 for both models). For robustness, we implemented the calibrated model (calibration slope 0.60; 95% CI 0.49-0.70) into COPE for predicting ICU admission. To predict the need for ICU admission – rather than historically observed ICU admission – COPE ignores the decreasing age effect after the age of 70 when making future predictions. Due to the weaker predictor effects, the discriminative ability of COPE was more moderate for predicting ICU admission than for predicting death in first wave patients below the age of 70 (Supplementary figure 3; AUC 0.66; 0.79). Although COPE overestimated ICU admission in 706 second wave patients below the age of 70, it was better able to identify the patients at high risk of needing IC in these patients (Figure 3; AUC 0.84; 0.81).
Model presentation
The resulting COPE model for predicting death as well as ICU admission within 28 days after hospital admission (formulas in Figure 4) are implemented as a publicly accessible web-based application https://mrpredicts.shinyapps.io/COPE/. For optimal transparency, the web application includes a detailed description of the derivation of COPE (Supplement 1), descriptions of the data that was used for development and validation of COPE, and calibration plots of temporal validation in separate hospitals.
DISCUSSION
We developed COPE for prediction of in-hospital death and need for intensive care when patients with suspected COVID-19 present at the Emergency Department. Developed in patient data from the first wave of the pandemic, based on six routinely measured predictors – age, respiratory rate, LDH, CRP, albumin and urea – COPE discriminated well and was well-calibrated in patients admitted to hospital in the second wave of the pandemic, both for predicting in-hospital death and for ICU admission.
External validation is considered optimal, that is validation of COPE in hospitals that were not present in the development data (10). However, the combination of temporal – in second wave patients – and geographic – in separate hospitals – validation is a strength of this study (11).
Although COPE already performed very well when validated in time and space, future research should focus on analyses of potential time trends not captured by the predictors, potential changes in predictor effects in time (interactions between predictors and time), and the impact of potential differences in patient case mix in countries other than the Netherlands (international validation). These case mix differences will primarily affect calibration, requiring an update of the model intercept.
In conclusion, a simple model, based on 6 routinely measured predictors in the ED, is well able to predict mortality and ICU admission for patients who present to the ED with suspected COVID-19. The model may help to inform patients and doctors when deciding on hospital admission.
Data Availability
According to data transfer agreements the data is not publicly available.
AUTHORS’ CONTRIBUTIONS
David van Klaveren, Hester Lingsma and Stephanie CE Klein Nagelvoort Schuit designed the study. Jelmer Alsma, Rob JCG Verdonschot, Dick TJJ Koning, Marlijn JA Kamps, Tom Dormans, Robert Stassen, Sebastiaan Weijer, Klaas-Sierk Arnold and Marieke Roelink were responsible for gathering the data. David van Klaveren analyzed the data and wrote the first draft of the paper. Alexandros Rekkas implemented the models into a web application. All authors contributed to writing the paper and approved the final version.
Supplement 1 Description COPE web application
Background and aim
The COVID-19 pandemic is putting extraordinary pressure on emergency departments (EDs). Clinical prediction models have the potential to support decision making about hospital admission, but currently available models were recently assessed to contain a high risk of bias. We aimed to develop a simple and valid model for predicting mortality and need for ICU in patients who are suspected to have COVID-19 when presenting at the ED.
Methods
For model development, we included patients that presented at the ED and were admitted to 4 large Dutch hospitals with suspected COVID-19 between March and August 2020, the first wave of the pandemic in the Netherlands. Patients being transferred from or to other hospitals were excluded since information on predictors or outcomes was missing. The outcomes of interest were death and admission to ICU within 28 days. Based on prior literature we included patient characteristics (sex, age, BMI), vital parameters (oxygen saturation, systolic blood pressure, heart rate, respiratory rate [RR], body temperature) and blood test values (C-reactive protein [CRP], lactic dehydrogenase [LDH], D-Dimer, leucocytes, lymphocytes, monocytes, neutrophils, eosinophils, MCV, albumin, bicarbonate, creatinine, sodium, urea), all measured at ED admission, as potential predictors. Further we included month of admission to capture changes in outcomes over time. Logistic regression was used to obtain predicted probabilities of death and of being admitted to the ICU, both within 28 days after admission. Model performance was assessed with temporal validation in patients who presented between September and December 2020 (second wave). We assessed discriminative ability with the area under the operator receiver characteristic curve (AUC) and calibration with calibration plots, calibration intercepts, and calibration slopes. We used multiple imputation to account for missing predictor values.
Results
The development data included 5,831 patients who presented and were admitted at the ED up until August 2020, of whom 629 (10.8%) died and 5,070 (86.9%) were discharged within 28 days after admission. A simple model – named COVID Outcome Prediction in the Emergency Department (COPE) – with linear age and logarithmic transforms of RR, CRP, LDH, Albumin and Urea captured most of the ability to predict death within 28 days. Patients who were admitted in the first month of the pandemic in the Netherlands had substantially increased risk of death (odds ratio 2.06; 95% confidence interval 1.68-2.52). COPE was well-calibrated and showed good discrimination for predicting death in 3,252 patients in the second wave (AUC in 4 hospitals: 0.82; 0.82; 0.79; 0.83). Admission to ICU was fully recorded for 2,633 first wave patients in 2 hospitals (214 ICU admissions within 28 days). The same predictors captured most of the ability to predict ICU admission within 28 days. However, after the age of 70, the probability of being admitted to the ICU was decreasing with age, probably reflecting the decision not to admit older patients to the ICU. To predict the need for ICU admission – rather than historically observed ICU admission – we kept a linear (decreasing) age effect after the age of 70 in the model, which will be ignored when making future predictions. COPE was well able to identify patients at high risk of needing IC in second wave patients below the age of 70 (AUC 0.84; 0.81), but overestimated ICU admission for low-risk patients. The models are implemented as a web-based application.
Conclusion
COPE, a simple tool based on 6 routinely measured predictors in the ED, is well able to predict mortality and ICU admission for patients who present to the ED with suspected COVID-19. COPE may help to inform patients and doctors when deciding on hospital admission.
ACKNOWLEDGMENTS
This work was supported by ZonMw (project number 10430 01 201 0019: Clinical prediction models for COVID-19: development, international validation and use) and the Patient-Centered Outcomes Research Institute (PCORI grant number ME-1606-35555: How Well Do Clinical Prediction Models (CPMs) Validate? A Large-Scale Evaluation of Cardiovascular Clinical Prediction Models).
We thank Noreen van der Linden and the Dutch Network of Acute Care (LNAZ) for support with gathering of the data.