Skip to main content
medRxiv
  • Home
  • About
  • Submit
  • ALERTS / RSS
Advanced Search

Development and external validation of prognostic models for COVID-19 to support risk stratification in secondary care

View ORCID ProfileNicola J Adderley, Thomas Taverner, View ORCID ProfileMalcolm Price, View ORCID ProfileChristopher Sainsbury, David Greenwood, View ORCID ProfileJoht Singh Chandan, View ORCID ProfileYemisi Takwoingi, View ORCID ProfileRashan Haniffa, Isaac Hosier, View ORCID ProfileCarly Welch, View ORCID ProfileDhruv Parekh, Suzy Gallier, View ORCID ProfileKrishna Gokhale, View ORCID ProfileAlastair K Denniston, View ORCID ProfileElizabeth Sapey, View ORCID ProfileKrishnarajah Nirantharakumar
doi: https://doi.org/10.1101/2021.01.25.21249942
Nicola J Adderley
1Institute of Applied Health Research, University of Birmingham, Edgbaston, Birmingham, B15 2TT, UK
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Nicola J Adderley
  • For correspondence: n.j.adderley@bham.ac.uk
Thomas Taverner
1Institute of Applied Health Research, University of Birmingham, Edgbaston, Birmingham, B15 2TT, UK
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Malcolm Price
1Institute of Applied Health Research, University of Birmingham, Edgbaston, Birmingham, B15 2TT, UK
2NIHR Birmingham Biomedical Research Centre, University Hospitals Birmingham NHS Foundation Trust and University of Birmingham, UK
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Malcolm Price
Christopher Sainsbury
1Institute of Applied Health Research, University of Birmingham, Edgbaston, Birmingham, B15 2TT, UK
3Department of Diabetes, Gartnavel General Hospital, NHS Greater Glasgow and Clyde, Glasgow, UK
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Christopher Sainsbury
David Greenwood
1Institute of Applied Health Research, University of Birmingham, Edgbaston, Birmingham, B15 2TT, UK
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Joht Singh Chandan
1Institute of Applied Health Research, University of Birmingham, Edgbaston, Birmingham, B15 2TT, UK
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Joht Singh Chandan
Yemisi Takwoingi
1Institute of Applied Health Research, University of Birmingham, Edgbaston, Birmingham, B15 2TT, UK
2NIHR Birmingham Biomedical Research Centre, University Hospitals Birmingham NHS Foundation Trust and University of Birmingham, UK
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Yemisi Takwoingi
Rashan Haniffa
4National Intensive Care Surveillance-MORU, Colombo, Sri Lanka
5Mahidol Oxford Tropical Medicine Research Unit, University of Oxford
6University College London Hospitals, London, United Kingdom
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Rashan Haniffa
Isaac Hosier
1Institute of Applied Health Research, University of Birmingham, Edgbaston, Birmingham, B15 2TT, UK
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Carly Welch
7Institute of Inflammation and Ageing, University of Birmingham, Birmingham, UK
8University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK
9Medical Research Council and Versus Arthritis Centre for Musculoskeletal Ageing Research, University of Birmingham and University of Nottingham, Birmingham and Nottingham, UK
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Carly Welch
Dhruv Parekh
7Institute of Inflammation and Ageing, University of Birmingham, Birmingham, UK
8University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Dhruv Parekh
Suzy Gallier
8University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Krishna Gokhale
1Institute of Applied Health Research, University of Birmingham, Edgbaston, Birmingham, B15 2TT, UK
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Krishna Gokhale
Alastair K Denniston
7Institute of Inflammation and Ageing, University of Birmingham, Birmingham, UK
10University College London/Moorfields Eye Hospital National Institute for Health Research Biomedical Research Centre
11Health Data Research UK
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Alastair K Denniston
Elizabeth Sapey
7Institute of Inflammation and Ageing, University of Birmingham, Birmingham, UK
8University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK
11Health Data Research UK
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Elizabeth Sapey
Krishnarajah Nirantharakumar
1Institute of Applied Health Research, University of Birmingham, Edgbaston, Birmingham, B15 2TT, UK
11Health Data Research UK
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Krishnarajah Nirantharakumar
  • Abstract
  • Full Text
  • Info/History
  • Metrics
  • Supplementary material
  • Data/Code
  • Preview PDF
Loading

Abstract

Objectives Existing UK prognostic models for patients admitted to hospital with COVID-19 are limited by reliance on comorbidities, which are under-recorded in secondary care, and lack of imaging data among the candidate predictors. Our aims were to develop and externally validate novel prognostic models for adverse outcomes (death, intensive therapy unit (ITU) admission) in UK secondary care; and externally validate the existing 4C score.

Design Candidate predictors included demographic variables, symptoms, physiological measures, imaging, laboratory tests. Final models used logistic regression with stepwise selection.

Setting Model development was performed in data from University Hospitals Birmingham (UHB). External validation was performed in the CovidCollab dataset.

Participants Patients with COVID-19 admitted to UHB January-August 2020 were included.

Main outcome measures Death and ITU admission within 28 days of admission.

Results 1040 patients with COVID-19 were included in the derivation cohort; 288 (28%) died and 183 (18%) were admitted to ITU within 28 days of admission. Area under the receiver operating curve (AUROC) for mortality was 0.791 (95%CI 0.761-0.822) in UHB and 0.767 (95%CI 0.754-0.780) in CovidCollab; AUROC for ITU admission was 0.906 (95%CI 0.883-0.929) in UHB and 0.811 (95%CI 0.795-0.828) in CovidCollab. Models showed good calibration. Addition of comorbidities to candidate predictors did not improve model performance. AUROC for the 4C score in the UHB dataset was 0.754 (95%CI 0.721-0.786).

Conclusions The novel prognostic models showed good discrimination and calibration in derivation and external validation datasets, and outperformed the existing 4C score. The models can be integrated into electronic medical records systems to calculate each individual patient’s probability of death or ITU admission at the time of hospital admission. Implementation of the models and clinical utility should be evaluated.

Strengths and limitations of this study

  • We developed novel prognostic models predicting mortality and ITU admission within 28 days of admission for patients hospitalised with COVID-19, using a large routinely collected dataset gathered at admission with a wide range of possible predictors (demographic variables, symptoms, physiological measures, imaging, laboratory test results).

  • These novel models showed good discrimination and calibration in both derivation and external validation cohorts, and outperformed the existing ISARIC model and 4C score in the derivation dataset. We found that addition of comorbidities to the set of candidate predictors included in model derivation did not improve model performance.

  • If integrated into hospital electronic medical records systems, the model algorithms will provide a predicted probability of mortality or ITU admission for each patient based on their individual data at, or close to, the time of admission, which will support clinicians’ decision making with regard to appropriate patient care pathways and triage. This information might also assist clinicians in explaining complex prognostic assessments and decisions to patients and their relatives.

  • A limitation of the study was that in the external validation cohort we were unable to examine all of the predictors included in the original full UHB model due to only a reduced set of candidate predictors being available in CovidCollab. Nevertheless, the reduced model performed well and the results suggest it may be applicable in a wide range of datasets where only a reduced set of predictor variables is available.

  • Furthermore, it was not possible to carry out stratified analysis by ethnicity as the UHB dataset contained too few patients in most of the strata, and no ethnicity data was available in the CovidCollab dataset.

Competing Interest Statement

NJA, ES, KN, MP, AD, CS, TT and YT report a grant from UKRI MRC during the conduct of the study. ES reports grants from National Institute for Health Research (NIHR), Wellcome Trust, MRC, Health Data Research UK (HDR-UK), British Lung Foundation, and Alpha 1 Foundation outside the submitted work. KN reports grants from MRC and HDR-UK outside the submitted work. DP reports grants from NIHR, MRC, and Chernakovsky Foundation outside the submitted work. All other authors have nothing to declare.

Funding Statement

This work was funded by the Medical Research Council UK Research and Innovation (MRC UKRI) (reference COV0306). The funder had no role in developing the research question or the study protocol.

Author Declarations

I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.

Yes

The details of the IRB/oversight body that provided approval or exemption for the research described are given below:

Ethical approval was provided by the East Midlands Derby REC (reference: 20/EM/0158) for the PIONEER Research Database (data from University Hospitals Birmingham). For CovidCollab data, local, regional, and national approvals were obtained from all participating sites. In the UK, this study was registered as clinical audit or service evaluation, with approval granted in line with local information governance policies, in line with assessment and guidance by the Health Research Authority. At the lead site (University Hospitals Birmingham NHS Trust) this study was registered as clinical audit (CARMS-15986). In other countries, local principal investigators were responsible for obtaining approvals in line with their local, regional, and national guidelines and recommendations. Only routinely collected data was collected and patient care was not altered by this study. Anonymised data was securely transferred to the Birmingham Centre for Prospective and Observational Studies (BiCOPS), University of Birmingham via REDCap. All sites were required to confirm that approvals were in place prior to being provided with logins; written data sharing agreements were arranged where requested by individual sites.

All necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived.

Yes

I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).

Yes

I have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.

Yes

Data Availability

No additional data available.

Copyright 
The copyright holder for this preprint is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. It is made available under a CC-BY-NC-ND 4.0 International license.
Back to top
PreviousNext
Posted January 30, 2021.
Download PDF

Supplementary Material

Data/Code
Email

Thank you for your interest in spreading the word about medRxiv.

NOTE: Your email address is requested solely to identify you as the sender of this article.

Enter multiple addresses on separate lines or separate them with commas.
Development and external validation of prognostic models for COVID-19 to support risk stratification in secondary care
(Your Name) has forwarded a page to you from medRxiv
(Your Name) thought you would like to see this page from the medRxiv website.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.
Share
Development and external validation of prognostic models for COVID-19 to support risk stratification in secondary care
Nicola J Adderley, Thomas Taverner, Malcolm Price, Christopher Sainsbury, David Greenwood, Joht Singh Chandan, Yemisi Takwoingi, Rashan Haniffa, Isaac Hosier, Carly Welch, Dhruv Parekh, Suzy Gallier, Krishna Gokhale, Alastair K Denniston, Elizabeth Sapey, Krishnarajah Nirantharakumar
medRxiv 2021.01.25.21249942; doi: https://doi.org/10.1101/2021.01.25.21249942
Digg logo Reddit logo Twitter logo Facebook logo Google logo LinkedIn logo Mendeley logo
Citation Tools
Development and external validation of prognostic models for COVID-19 to support risk stratification in secondary care
Nicola J Adderley, Thomas Taverner, Malcolm Price, Christopher Sainsbury, David Greenwood, Joht Singh Chandan, Yemisi Takwoingi, Rashan Haniffa, Isaac Hosier, Carly Welch, Dhruv Parekh, Suzy Gallier, Krishna Gokhale, Alastair K Denniston, Elizabeth Sapey, Krishnarajah Nirantharakumar
medRxiv 2021.01.25.21249942; doi: https://doi.org/10.1101/2021.01.25.21249942

Citation Manager Formats

  • BibTeX
  • Bookends
  • EasyBib
  • EndNote (tagged)
  • EndNote 8 (xml)
  • Medlars
  • Mendeley
  • Papers
  • RefWorks Tagged
  • Ref Manager
  • RIS
  • Zotero
  • Tweet Widget
  • Facebook Like
  • Google Plus One

Subject Area

  • Public and Global Health
Subject Areas
All Articles
  • Addiction Medicine (174)
  • Allergy and Immunology (419)
  • Anesthesia (97)
  • Cardiovascular Medicine (894)
  • Dentistry and Oral Medicine (166)
  • Dermatology (101)
  • Emergency Medicine (257)
  • Endocrinology (including Diabetes Mellitus and Metabolic Disease) (404)
  • Epidemiology (8745)
  • Forensic Medicine (4)
  • Gastroenterology (403)
  • Genetic and Genomic Medicine (1845)
  • Geriatric Medicine (175)
  • Health Economics (386)
  • Health Informatics (1282)
  • Health Policy (642)
  • Health Systems and Quality Improvement (489)
  • Hematology (206)
  • HIV/AIDS (387)
  • Infectious Diseases (except HIV/AIDS) (10521)
  • Intensive Care and Critical Care Medicine (564)
  • Medical Education (193)
  • Medical Ethics (52)
  • Nephrology (216)
  • Neurology (1744)
  • Nursing (100)
  • Nutrition (264)
  • Obstetrics and Gynecology (342)
  • Occupational and Environmental Health (457)
  • Oncology (959)
  • Ophthalmology (279)
  • Orthopedics (107)
  • Otolaryngology (174)
  • Pain Medicine (117)
  • Palliative Medicine (41)
  • Pathology (262)
  • Pediatrics (553)
  • Pharmacology and Therapeutics (264)
  • Primary Care Research (218)
  • Psychiatry and Clinical Psychology (1836)
  • Public and Global Health (3963)
  • Radiology and Imaging (649)
  • Rehabilitation Medicine and Physical Therapy (340)
  • Respiratory Medicine (532)
  • Rheumatology (214)
  • Sexual and Reproductive Health (178)
  • Sports Medicine (165)
  • Surgery (196)
  • Toxicology (37)
  • Transplantation (105)
  • Urology (78)