Skip to main content
medRxiv
  • Home
  • About
  • Submit
  • ALERTS / RSS
Advanced Search

Clinical Prediction Models for Primary Prevention of Cardiovascular Disease: Validity in Independent Cohorts

Gaurav Gulati, Riley J Brazil, Jason Nelson, David van Klaveren, Christine M. Lundquist, Jinny G. Park, Hannah McGinnes, Ewout W. Steyerberg, Ben Van Calster, Benjamin S. Wessler, David M. Kent
doi: https://doi.org/10.1101/2021.01.31.21250871
Gaurav Gulati
1Predictive Analytics and Comparative Effectiveness (PACE) Center, Institute for Clinical Research and Health Policy Studies (ICRHPS), Tufts Medical Center, Boston, MA
2Division of Cardiology, Tufts Medical Center, Boston, MA
MD
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Riley J Brazil
1Predictive Analytics and Comparative Effectiveness (PACE) Center, Institute for Clinical Research and Health Policy Studies (ICRHPS), Tufts Medical Center, Boston, MA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Jason Nelson
1Predictive Analytics and Comparative Effectiveness (PACE) Center, Institute for Clinical Research and Health Policy Studies (ICRHPS), Tufts Medical Center, Boston, MA
MPH
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
David van Klaveren
1Predictive Analytics and Comparative Effectiveness (PACE) Center, Institute for Clinical Research and Health Policy Studies (ICRHPS), Tufts Medical Center, Boston, MA
3Department of Public Health, Erasmus University Medical Center, Rotterdam
PhD
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Christine M. Lundquist
1Predictive Analytics and Comparative Effectiveness (PACE) Center, Institute for Clinical Research and Health Policy Studies (ICRHPS), Tufts Medical Center, Boston, MA
MPH
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Jinny G. Park
1Predictive Analytics and Comparative Effectiveness (PACE) Center, Institute for Clinical Research and Health Policy Studies (ICRHPS), Tufts Medical Center, Boston, MA
MPH
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Hannah McGinnes
1Predictive Analytics and Comparative Effectiveness (PACE) Center, Institute for Clinical Research and Health Policy Studies (ICRHPS), Tufts Medical Center, Boston, MA
MPH
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Ewout W. Steyerberg
4Department of Biomedical Data Sciences, Leiden University Medical Centre, Leiden, Netherlands
PhD
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Ben Van Calster
5KU Leuven, Department of Development and Regeneration, Leuven, Belgium
PhD
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Benjamin S. Wessler
1Predictive Analytics and Comparative Effectiveness (PACE) Center, Institute for Clinical Research and Health Policy Studies (ICRHPS), Tufts Medical Center, Boston, MA
2Division of Cardiology, Tufts Medical Center, Boston, MA
MD MS
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
David M. Kent
1Predictive Analytics and Comparative Effectiveness (PACE) Center, Institute for Clinical Research and Health Policy Studies (ICRHPS), Tufts Medical Center, Boston, MA
MD MS
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • For correspondence: dkent1@tuftsmedicalcenter.org
  • Abstract
  • Full Text
  • Info/History
  • Metrics
  • Supplementary material
  • Data/Code
  • Preview PDF
Loading

Abstract

Background Clinical prediction models (CPMs) are used to inform treatment decisions for the primary prevention of cardiovascular disease. We aimed to assess the performance of such CPMs in fully independent cohorts.

Methods and Results 63 models predicting outcomes for patients at risk of cardiovascular disease from the Tufts PACE CPM Registry were selected for external validation on publicly available data from up to 4 broadly inclusive primary prevention clinical trials. For each CPM-trial pair, we assessed model discrimination, calibration, and net benefit. Results were stratified based on the relatedness of derivation and validation cohorts, and net benefit was reassessed after updating model intercept, slope, or complete re-estimation. The median c statistic of the CPMs decreased from 0.77 (IQR 0.72-0.78) in the derivation cohorts to 0.63 (IQR 0.58-0.66) when externally validated. The validation c-statistic was higher when derivation and validation cohorts were considered related than when they were distantly related (0.67 vs 0.60, p < 0.001). The calibration slope was also higher in related cohorts than distantly related cohorts (0.69 vs 0.58, p < 0.001). Net benefit analysis suggested substantial likelihood of harm when models were externally applied, but this likelihood decreased after model updating.

Conclusions Discrimination and calibration decrease significantly when CPMs for primary prevention of cardiovascular disease are tested in external populations, particularly when the population is only distantly related to the derivation population. Poorly calibrated predictions lead to poor decision making. Model updating can reduce the likelihood of harmful decision making, and is needed to realize the full potential of risk-based decision making in new settings.

Competing Interest Statement

The authors have declared no competing interest.

Clinical Trial

N/A not a clinical trial

Funding Statement

Research reported in this work was funded through a Patient-Centered Outcomes Research Institute (PCORI) Award (ME-1606-35555).

Author Declarations

I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.

Yes

The details of the IRB/oversight body that provided approval or exemption for the research described are given below:

Tufts Health Sciences IRB # 12461

All necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived.

Yes

I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).

Yes

I have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.

Yes

Data Availability

The Tufts Predictive Analytics and Comparative Effectiveness (PACE) CPM Registry

http://pacecpmregistry.org/

Copyright 
The copyright holder for this preprint is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. It is made available under a CC-BY-NC-ND 4.0 International license.
Back to top
PreviousNext
Posted February 03, 2021.
Download PDF

Supplementary Material

Data/Code
Email

Thank you for your interest in spreading the word about medRxiv.

NOTE: Your email address is requested solely to identify you as the sender of this article.

Enter multiple addresses on separate lines or separate them with commas.
Clinical Prediction Models for Primary Prevention of Cardiovascular Disease: Validity in Independent Cohorts
(Your Name) has forwarded a page to you from medRxiv
(Your Name) thought you would like to see this page from the medRxiv website.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.
Share
Clinical Prediction Models for Primary Prevention of Cardiovascular Disease: Validity in Independent Cohorts
Gaurav Gulati, Riley J Brazil, Jason Nelson, David van Klaveren, Christine M. Lundquist, Jinny G. Park, Hannah McGinnes, Ewout W. Steyerberg, Ben Van Calster, Benjamin S. Wessler, David M. Kent
medRxiv 2021.01.31.21250871; doi: https://doi.org/10.1101/2021.01.31.21250871
Reddit logo Twitter logo Facebook logo LinkedIn logo Mendeley logo
Citation Tools
Clinical Prediction Models for Primary Prevention of Cardiovascular Disease: Validity in Independent Cohorts
Gaurav Gulati, Riley J Brazil, Jason Nelson, David van Klaveren, Christine M. Lundquist, Jinny G. Park, Hannah McGinnes, Ewout W. Steyerberg, Ben Van Calster, Benjamin S. Wessler, David M. Kent
medRxiv 2021.01.31.21250871; doi: https://doi.org/10.1101/2021.01.31.21250871

Citation Manager Formats

  • BibTeX
  • Bookends
  • EasyBib
  • EndNote (tagged)
  • EndNote 8 (xml)
  • Medlars
  • Mendeley
  • Papers
  • RefWorks Tagged
  • Ref Manager
  • RIS
  • Zotero
  • Tweet Widget
  • Facebook Like
  • Google Plus One

Subject Area

  • Cardiovascular Medicine
Subject Areas
All Articles
  • Addiction Medicine (269)
  • Allergy and Immunology (549)
  • Anesthesia (135)
  • Cardiovascular Medicine (1749)
  • Dentistry and Oral Medicine (238)
  • Dermatology (172)
  • Emergency Medicine (310)
  • Endocrinology (including Diabetes Mellitus and Metabolic Disease) (654)
  • Epidemiology (10785)
  • Forensic Medicine (8)
  • Gastroenterology (584)
  • Genetic and Genomic Medicine (2935)
  • Geriatric Medicine (286)
  • Health Economics (531)
  • Health Informatics (1919)
  • Health Policy (833)
  • Health Systems and Quality Improvement (743)
  • Hematology (290)
  • HIV/AIDS (627)
  • Infectious Diseases (except HIV/AIDS) (12501)
  • Intensive Care and Critical Care Medicine (685)
  • Medical Education (299)
  • Medical Ethics (86)
  • Nephrology (323)
  • Neurology (2786)
  • Nursing (150)
  • Nutrition (431)
  • Obstetrics and Gynecology (556)
  • Occupational and Environmental Health (597)
  • Oncology (1458)
  • Ophthalmology (441)
  • Orthopedics (172)
  • Otolaryngology (255)
  • Pain Medicine (190)
  • Palliative Medicine (56)
  • Pathology (380)
  • Pediatrics (865)
  • Pharmacology and Therapeutics (362)
  • Primary Care Research (334)
  • Psychiatry and Clinical Psychology (2633)
  • Public and Global Health (5342)
  • Radiology and Imaging (1004)
  • Rehabilitation Medicine and Physical Therapy (595)
  • Respiratory Medicine (724)
  • Rheumatology (329)
  • Sexual and Reproductive Health (289)
  • Sports Medicine (278)
  • Surgery (327)
  • Toxicology (47)
  • Transplantation (149)
  • Urology (125)