TY - JOUR T1 - External Validations of Cardiovascular Clinical Prediction Models: A Large-scale Review of the Literature JF - medRxiv DO - 10.1101/2021.01.19.21250110 SP - 2021.01.19.21250110 AU - Benjamin S. Wessler AU - Jason Nelson AU - Jinny G. Park AU - Hannah McGinnes AU - Gaurav Gulati AU - Riley Brazil AU - Ben Van Calster AU - D. van Klaveren AU - Esmee Venema AU - Ewout Steyerberg AU - Jessica K. Paulus AU - David M. Kent Y1 - 2021/01/01 UR - http://medrxiv.org/content/early/2021/01/21/2021.01.19.21250110.abstract N2 - Background There are many clinical prediction models (CPMs) available to inform treatment decisions for patients with cardiovascular disease. However, the extent to which they have been externally tested and how well they generally perform has not been broadly evaluated.Methods A SCOPUS citation search was run on March 22, 2017 to identify external validations of cardiovascular CPMs in the Tufts PACE CPM Registry. We assessed the extent of external validation, performance heterogeneity across databases, and explored factors associated with model performance, including a global assessment of the clinical relatedness between the derivation and validation data.Results 2030 external validations of 1382 CPMs were identified. 807 (58%) of the CPMs in the Registry have never been externally validated. On average there were 1.5 validations per CPM (range 0-94). The median external validation AUC was 0.73 (25th −75th percentile [IQR] 0.66, 0.79), representing a median percent decrease in discrimination of −11.1% (IQR −32.4%, +2.7%) compared to performance on derivation data. 81% (n = 1333) of validations reporting AUC showed discrimination below that reported in the derivation dataset. 53% (n = 983) of the validations report some measure of CPM calibration. For CPMs evaluated more than once, there was typically a large range of performance. Of 1702 validations classified by relatedness, the percent change in discrimination was −3.7% (IQR −13.2, 3.1) for ‘closely related’ validations (n=123), −9.0 (IQR −27.6, 3.9) for ‘related validations’ (n=862) and −17.2% (IQR −42.3, 0) for ‘distantly related’ validations (n=717) (p<0.001).Conclusion Many published cardiovascular CPMs have never been externally validated and for those that have, apparent performance during development is often overly optimistic. A single external validation appears insufficient to broadly understand the performance heterogeneity across different settings.Competing Interest StatementThe authors have declared no competing interest.Clinical TrialN/A not a clinical trialFunding StatementResearch reported in this work was funded through a Patient-Centered Outcomes Research Institute (PCORI) Award (ME-1606-35555).Author DeclarationsI confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.YesThe details of the IRB/oversight body that provided approval or exemption for the research described are given below:Tufts Health Sciences IRB Study number #12461All necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived.YesI understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).Yes I have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.YesThe authors confirm that the data supporting the findings of this study are available within the article. ER -