Skip to main content
medRxiv
  • Home
  • About
  • Submit
  • ALERTS / RSS
Advanced Search

Regional performance variation in external validation of four prediction models for severity of COVID-19 at hospital admission: An observational multi-centre cohort study

Kristin E. Wickstrøm, Valeria Vitelli, Ewan Carr, Aleksander R. Holten, Rebecca Bendayan, Andrew H. Reiner, View ORCID ProfileDaniel Bean, Tom Searle, Anthony Shek, Zeljko Kraljevic, View ORCID ProfileJames Teo, Richard Dobson, Kristian Tonby, Alvaro Köhn- Luque, View ORCID ProfileErik K. Amundsen
doi: https://doi.org/10.1101/2021.03.26.21254390
Kristin E. Wickstrøm
1Department of Medical Biochemistry, Blood Cell Research Group, Oslo University Hospital, Oslo, Norway
2Institute of Clinical Medicine, University of Oslo, Oslo, Norway
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Valeria Vitelli
3Oslo Centre for Biostatistics and Epidemiology, Faculty of Medicine, University of Oslo, Oslo, Norway
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Ewan Carr
4Department of Biostatistics and Health Informatics, Institute of Psychiatry, Psychology and Neuroscience, King’s College London, London, U.K.
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Aleksander R. Holten
2Institute of Clinical Medicine, University of Oslo, Oslo, Norway
5Department of Acute Medicine, Oslo University Hospital and Institute of Clinical Medicine, University of Oslo, Oslo, Norway
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Rebecca Bendayan
4Department of Biostatistics and Health Informatics, Institute of Psychiatry, Psychology and Neuroscience, King’s College London, London, U.K.
6NIHR Biomedical Research Centre at South London and Maudsley NHS Foundation Trust and King’s College London, London, U.K.
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Andrew H. Reiner
7Oslo Centre for Biostatistics and Epidemiology, Oslo University Hospital, Oslo, Norway
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Daniel Bean
4Department of Biostatistics and Health Informatics, Institute of Psychiatry, Psychology and Neuroscience, King’s College London, London, U.K.
8Health Data Research UK London, University College London, London, U.K.
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Daniel Bean
Tom Searle
4Department of Biostatistics and Health Informatics, Institute of Psychiatry, Psychology and Neuroscience, King’s College London, London, U.K.
6NIHR Biomedical Research Centre at South London and Maudsley NHS Foundation Trust and King’s College London, London, U.K.
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Anthony Shek
9Department of Clinical Neuroscience, Institute of Psychiatry, Psychology and Neuroscience, King’s College London, London, U.K.
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Zeljko Kraljevic
4Department of Biostatistics and Health Informatics, Institute of Psychiatry, Psychology and Neuroscience, King’s College London, London, U.K.
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
James Teo
9Department of Clinical Neuroscience, Institute of Psychiatry, Psychology and Neuroscience, King’s College London, London, U.K.
10King’s College Hospital NHS Foundation Trust, London, U.K.
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for James Teo
Richard Dobson
4Department of Biostatistics and Health Informatics, Institute of Psychiatry, Psychology and Neuroscience, King’s College London, London, U.K.
6NIHR Biomedical Research Centre at South London and Maudsley NHS Foundation Trust and King’s College London, London, U.K.
8Health Data Research UK London, University College London, London, U.K.
11Institute of Health Informatics, University College London, London, U.K.
12NIHR Biomedical Research Centre at University College London Hospitals NHS Foundation Trust, London, U.K.
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Kristian Tonby
2Institute of Clinical Medicine, University of Oslo, Oslo, Norway
13Dept. of Infectious diseases, Oslo University Hospital, Oslo, Norway
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Alvaro Köhn- Luque
3Oslo Centre for Biostatistics and Epidemiology, Faculty of Medicine, University of Oslo, Oslo, Norway
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Erik K. Amundsen
1Department of Medical Biochemistry, Blood Cell Research Group, Oslo University Hospital, Oslo, Norway
14Oslo Metropolitan University, Department of Life Sciences and Health
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Erik K. Amundsen
  • For correspondence: uxamue@ous-hf.no
  • Abstract
  • Full Text
  • Info/History
  • Metrics
  • Supplementary material
  • Data/Code
  • Preview PDF
Loading

Abstract

Background Several prediction models for coronavirus disease-19 (COVID-19) have been published. Prediction models should be externally validated to assess their performance before implementation. This observational cohort study aimed to validate published models of severity for hospitalized patients with COVID-19 using clinical and laboratory predictors.

Methods Prediction models fitting relevant inclusion criteria were chosen for validation. The outcome was either mortality or a composite outcome of mortality and ICU admission (severe disease). 1295 patients admitted with symptoms of COVID-19 at Kings Cross Hospital (KCH) in London, United Kingdom, and 307 patients at Oslo University Hospital (OUH) in Oslo, Norway were included. The performance of the models was assessed in terms of discrimination and calibration.

Results We identified two models for prediction of mortality (referred to as Xie and Zhang1) and two models for prediction of severe disease (Allenbach and Zhang2).

The performance of the models was variable. For prediction of mortality Xie had good discrimination at OUH with an area under the receiver-operating characteristic (AUROC) 0.87 [95 % confidence interval (CI) 0.79-0.95] and acceptable discrimination at KCH, AUROC 0.79 [0.76-0.82]. In prediction of severe disease, Allenbach had acceptable discrimination (OUH AUROC 0.81 [0.74-0.88] and KCH AUROC 0.72 [0.68-0.75]). The Zhang models had moderate to poor discrimination. Initial calibration was poor for all models but improved with recalibration.

Conclusions The performance of the four prediction models was variable. The Xie model had the best discrimination for mortality, while the Allenbach model had acceptable results for prediction of severe disease.

Competing Interest Statement

JTHT received research support and funding from InnovateUK, Bristol-Myers-Squibb, iRhythm Technologies, and holds shares <5,000 GBP in Glaxo Smithkline and Biogen.

Funding Statement

DMB is funded by a UKRI Innovation Fellowship as part of Health Data Research UK MR/S00310X/1 (https://www.hdruk.ac.uk). RB is funded in part by grant MR/R016372/1 for the Kings College London MRC Skills Development Fellowship programme funded by the UK Medical Research Council (MRC, https://mrc.ukri.org) and by grant IS-BRC-1215-20018 for the National Institute for Health Research (NIHR, https://www.nihr.ac.uk) Biomedical Research Centre at South London and Maudsley NHS Foundation Trust and Kings College London. RJBD is supported by: (1) NIHR Biomedical Research Centre at South London and Maudsley NHS Foundation Trust and Kings College London, London, U.K. (2) Health Data Research UK, which is funded by the UK Medical Research Council, Engineering and Physical Sciences Research Council, Economic and Social Research Council, Department of Health and Social Care (England), Chief Scientist Office of the Scottish Government Health and Social Care Directorates, Health and Social Care Research and Development Division (Welsh Government), Public Health Agency (Northern Ireland), British Heart Foundation and Wellcome Trust. (3) The BigData@Heart Consortium, funded by the Innovative Medicines Initiative-2 Joint Undertaking under grant agreement No. 116074. This Joint Undertaking receives support from the European Union Horizon 2020 research and innovation program and EFPIA; it is chaired by DE Grobbee and SD Anker, partnering with 20 academic and industry partners and ESC. (4) The National Institute for Health Research University College London Hospitals Biomedical Research Centre. (5) National Institute for Health Research (NIHR) Biomedical Research Centre at South London and Maudsley NHS Foundation Trust and Kings College London. (5) The UK Research and Innovation London Medical Imaging & Artificial Intelligence Centre for Value Based Healthcare (6) the National Institute for Health Research (NIHR) Applied Research Collaboration South London (NIHR ARC South London) at Kings College Hospital NHS Foundation Trust.

Author Declarations

I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.

Yes

The details of the IRB/oversight body that provided approval or exemption for the research described are given below:

The OUH project protocol was approved by the Regional Ethical Committee of South East Norway (Reference 137045). All patients with confirmed COVID-19 were included in the quality registry COVID19 OUS, approved by the data protection officer (Reference 20/08822). Informed consent was waived because of the strictly observational nature of the project. The KCH project operated under London South East Research Ethics Committee (reference 18/LO/2048) approval granted to the Kings Electronic Records Research Interface (KERRI); specific work on COVID-19 research was reviewed with expert patient input on a virtual committee with Caldicott Guardian oversight.

All necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived.

Yes

I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).

Yes

I have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.

Yes

Data Availability

Code and pre-trained models are available at https://github.com/ocbe.uio/covid-19 model validation and openly shared for testing in other COVID-19 datasets. KCH: Source text from patient records used at all sites in the study will not be available due to inability to safely fully anonymise up to the Information Commissioner Office (ICO) standards and would be likely to contain strong identifiers (e.g. names, postcodes) and highly sensitive data (e.g. diagnoses). A subset of the KCH dataset limited to anonymisable information (e.g. only SNOMED codes and aggregated demographics) is available on request to researchers with suitable training in information governance and human confidentiality protocols subject to approval by the Kings College Hospital Information Governance committee; applications for research access should be sent to kch-tr.cogstackrequests{at}nhs.net. This dataset cannot be released publicly due to the risk of re-identification of such granular individual-level data, as determined by the Kings College Hospital Caldicott Guardian. OUH: The OUH dataset cannot be released publicly due to the risk of reidentification of such granular individual-level data. Researchers can contact Erik Koldberg Amundsen for any enquiries (uxamue@ous-hf.no)

Copyright 
The copyright holder for this preprint is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. All rights reserved. No reuse allowed without permission.
Back to top
PreviousNext
Posted March 26, 2021.
Download PDF

Supplementary Material

Data/Code
Email

Thank you for your interest in spreading the word about medRxiv.

NOTE: Your email address is requested solely to identify you as the sender of this article.

Enter multiple addresses on separate lines or separate them with commas.
Regional performance variation in external validation of four prediction models for severity of COVID-19 at hospital admission: An observational multi-centre cohort study
(Your Name) has forwarded a page to you from medRxiv
(Your Name) thought you would like to see this page from the medRxiv website.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.
Share
Regional performance variation in external validation of four prediction models for severity of COVID-19 at hospital admission: An observational multi-centre cohort study
Kristin E. Wickstrøm, Valeria Vitelli, Ewan Carr, Aleksander R. Holten, Rebecca Bendayan, Andrew H. Reiner, Daniel Bean, Tom Searle, Anthony Shek, Zeljko Kraljevic, James Teo, Richard Dobson, Kristian Tonby, Alvaro Köhn- Luque, Erik K. Amundsen
medRxiv 2021.03.26.21254390; doi: https://doi.org/10.1101/2021.03.26.21254390
Reddit logo Twitter logo Facebook logo LinkedIn logo Mendeley logo
Citation Tools
Regional performance variation in external validation of four prediction models for severity of COVID-19 at hospital admission: An observational multi-centre cohort study
Kristin E. Wickstrøm, Valeria Vitelli, Ewan Carr, Aleksander R. Holten, Rebecca Bendayan, Andrew H. Reiner, Daniel Bean, Tom Searle, Anthony Shek, Zeljko Kraljevic, James Teo, Richard Dobson, Kristian Tonby, Alvaro Köhn- Luque, Erik K. Amundsen
medRxiv 2021.03.26.21254390; doi: https://doi.org/10.1101/2021.03.26.21254390

Citation Manager Formats

  • BibTeX
  • Bookends
  • EasyBib
  • EndNote (tagged)
  • EndNote 8 (xml)
  • Medlars
  • Mendeley
  • Papers
  • RefWorks Tagged
  • Ref Manager
  • RIS
  • Zotero
  • Tweet Widget
  • Facebook Like
  • Google Plus One

Subject Area

  • Infectious Diseases (except HIV/AIDS)
Subject Areas
All Articles
  • Addiction Medicine (243)
  • Allergy and Immunology (524)
  • Anesthesia (125)
  • Cardiovascular Medicine (1432)
  • Dentistry and Oral Medicine (219)
  • Dermatology (158)
  • Emergency Medicine (292)
  • Endocrinology (including Diabetes Mellitus and Metabolic Disease) (589)
  • Epidemiology (10326)
  • Forensic Medicine (6)
  • Gastroenterology (533)
  • Genetic and Genomic Medicine (2652)
  • Geriatric Medicine (255)
  • Health Economics (499)
  • Health Informatics (1744)
  • Health Policy (791)
  • Health Systems and Quality Improvement (680)
  • Hematology (269)
  • HIV/AIDS (570)
  • Infectious Diseases (except HIV/AIDS) (12116)
  • Intensive Care and Critical Care Medicine (650)
  • Medical Education (275)
  • Medical Ethics (83)
  • Nephrology (291)
  • Neurology (2478)
  • Nursing (145)
  • Nutrition (381)
  • Obstetrics and Gynecology (496)
  • Occupational and Environmental Health (569)
  • Oncology (1332)
  • Ophthalmology (403)
  • Orthopedics (152)
  • Otolaryngology (239)
  • Pain Medicine (172)
  • Palliative Medicine (51)
  • Pathology (344)
  • Pediatrics (785)
  • Pharmacology and Therapeutics (334)
  • Primary Care Research (296)
  • Psychiatry and Clinical Psychology (2417)
  • Public and Global Health (5029)
  • Radiology and Imaging (900)
  • Rehabilitation Medicine and Physical Therapy (533)
  • Respiratory Medicine (687)
  • Rheumatology (309)
  • Sexual and Reproductive Health (257)
  • Sports Medicine (246)
  • Surgery (300)
  • Toxicology (45)
  • Transplantation (141)
  • Urology (108)