Skip to main content
medRxiv
  • Home
  • About
  • Submit
  • ALERTS / RSS
Advanced Search

Accuracy of US CDC COVID-19 Forecasting Models

View ORCID ProfileAviral Chharia, Govind Jeevan, Rajat Aayush Jha, Meng Liu, Jonathan M Berman, View ORCID ProfileChristin Glorioso
doi: https://doi.org/10.1101/2022.04.20.22274097
Aviral Chharia
1Academics for the Future of Science, Cambridge, MA 02139, USA
2Data Informatics Center for Epidemiology, PathCheck Foundation, Cambridge, MA, USA
3Mechanical Engineering Department, Thapar Institute of Engineering and Technology, Patiala, PB 147004, India
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Aviral Chharia
Govind Jeevan
1Academics for the Future of Science, Cambridge, MA 02139, USA
2Data Informatics Center for Epidemiology, PathCheck Foundation, Cambridge, MA, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Rajat Aayush Jha
1Academics for the Future of Science, Cambridge, MA 02139, USA
2Data Informatics Center for Epidemiology, PathCheck Foundation, Cambridge, MA, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Meng Liu
1Academics for the Future of Science, Cambridge, MA 02139, USA
4Department of Industrial and Manufacturing Engineering, Penn State University, PA 16802, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Jonathan M Berman
1Academics for the Future of Science, Cambridge, MA 02139, USA
5Department of Basic Science, New York Institute of Technology College of Osteopathic Medicine at Arkansas State University, Jonesboro, Ar, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Christin Glorioso
1Academics for the Future of Science, Cambridge, MA 02139, USA
2Data Informatics Center for Epidemiology, PathCheck Foundation, Cambridge, MA, USA
6Department of Anatomy, University of California, San Francisco, San Francisco, CA 94143, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Christin Glorioso
  • For correspondence: gloriosoca@gmail.com
  • Abstract
  • Full Text
  • Info/History
  • Metrics
  • Data/Code
  • Preview PDF
Loading

ABSTRACT

Accurate predictive modeling of pandemics is essential for optimally distributing resources and setting policy. Dozens of case predictions models have been proposed but their accuracy over time and by model type remains unclear. In this study, we analyze all US CDC COVID-19 forecasting models, by first categorizing them and then calculating their mean absolute percent error, both wave-wise and on the complete timeline. We compare their estimates to government-reported case numbers, one another, as well as two baseline models wherein case counts remain static or follow a simple linear trend. The comparison reveals that more than one-third of models fail to outperform a simple static case baseline and two-thirds fail to outperform a simple linear trend forecast. A wave-by-wave comparison of models revealed that no overall modeling approach was superior to others, including ensemble models, and error in modeling has increased over time during the pandemic. This study raises concerns about hosting these models on official public platforms of health organizations including the US-CDC which risks giving them an official imprimatur and further raising concerns if utilized to formulate policy. By offering a universal evaluation method for pandemic forecasting models, we expect this work to serve as the starting point towards the development of more sophisticated models.

Competing Interest Statement

The authors have declared no competing interest.

Funding Statement

This study did not receive any funding.

Author Declarations

I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.

Yes

I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.

Yes

I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).

Yes

I have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.

Yes

Data Availability

The data used in this study is publicly available from the COVID-19 Data Repository, maintained by the Center for Systems Science and Engineering (CSSE) at Johns Hopkins University. The data for the predicted case counts, of all the models, is obtained from the data repository for the COVID-19 Forecast Hub, the data source for the official US-CDC COVID-19 forecasting page. The source code that supports the findings of this research is available from the corresponding author upon request.

https://github.com/CSSEGISandData/COVID-19/tree/master/csse_covid_19_data/csse_covid_19_daily_reports_us

https://github.com/reichlab/covid19-forecast-hub

Copyright 
The copyright holder for this preprint is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. It is made available under a CC-BY-NC-ND 4.0 International license.
Back to top
PreviousNext
Posted April 21, 2022.
Download PDF
Data/Code
Email

Thank you for your interest in spreading the word about medRxiv.

NOTE: Your email address is requested solely to identify you as the sender of this article.

Enter multiple addresses on separate lines or separate them with commas.
Accuracy of US CDC COVID-19 Forecasting Models
(Your Name) has forwarded a page to you from medRxiv
(Your Name) thought you would like to see this page from the medRxiv website.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.
Share
Accuracy of US CDC COVID-19 Forecasting Models
Aviral Chharia, Govind Jeevan, Rajat Aayush Jha, Meng Liu, Jonathan M Berman, Christin Glorioso
medRxiv 2022.04.20.22274097; doi: https://doi.org/10.1101/2022.04.20.22274097
Reddit logo Twitter logo Facebook logo LinkedIn logo Mendeley logo
Citation Tools
Accuracy of US CDC COVID-19 Forecasting Models
Aviral Chharia, Govind Jeevan, Rajat Aayush Jha, Meng Liu, Jonathan M Berman, Christin Glorioso
medRxiv 2022.04.20.22274097; doi: https://doi.org/10.1101/2022.04.20.22274097

Citation Manager Formats

  • BibTeX
  • Bookends
  • EasyBib
  • EndNote (tagged)
  • EndNote 8 (xml)
  • Medlars
  • Mendeley
  • Papers
  • RefWorks Tagged
  • Ref Manager
  • RIS
  • Zotero
  • Tweet Widget
  • Facebook Like
  • Google Plus One

Subject Area

  • Public and Global Health
Subject Areas
All Articles
  • Addiction Medicine (280)
  • Allergy and Immunology (580)
  • Anesthesia (141)
  • Cardiovascular Medicine (1962)
  • Dentistry and Oral Medicine (253)
  • Dermatology (187)
  • Emergency Medicine (335)
  • Endocrinology (including Diabetes Mellitus and Metabolic Disease) (703)
  • Epidemiology (11123)
  • Forensic Medicine (8)
  • Gastroenterology (629)
  • Genetic and Genomic Medicine (3196)
  • Geriatric Medicine (310)
  • Health Economics (567)
  • Health Informatics (2049)
  • Health Policy (864)
  • Health Systems and Quality Improvement (789)
  • Hematology (310)
  • HIV/AIDS (685)
  • Infectious Diseases (except HIV/AIDS) (12742)
  • Intensive Care and Critical Care Medicine (708)
  • Medical Education (318)
  • Medical Ethics (92)
  • Nephrology (337)
  • Neurology (3004)
  • Nursing (165)
  • Nutrition (465)
  • Obstetrics and Gynecology (589)
  • Occupational and Environmental Health (614)
  • Oncology (1561)
  • Ophthalmology (478)
  • Orthopedics (186)
  • Otolaryngology (266)
  • Pain Medicine (202)
  • Palliative Medicine (57)
  • Pathology (403)
  • Pediatrics (914)
  • Pharmacology and Therapeutics (386)
  • Primary Care Research (355)
  • Psychiatry and Clinical Psychology (2799)
  • Public and Global Health (5613)
  • Radiology and Imaging (1101)
  • Rehabilitation Medicine and Physical Therapy (637)
  • Respiratory Medicine (764)
  • Rheumatology (341)
  • Sexual and Reproductive Health (315)
  • Sports Medicine (289)
  • Surgery (348)
  • Toxicology (48)
  • Transplantation (159)
  • Urology (133)