TY - JOUR T1 - Large-scale validation of the Prediction model Risk Of Bias ASsessment Tool (PROBAST) using a short form: high risk of bias models show poorer discrimination JF - medRxiv DO - 10.1101/2021.01.20.21250183 SP - 2021.01.20.21250183 AU - Esmee Venema AU - Benjamin S Wessler AU - Jessica K Paulus AU - Rehab Salah AU - Gowri Raman AU - Lester Y Leung AU - Benjamin C Koethe AU - Jason Nelson AU - Jinny G Park AU - David van Klaveren AU - Ewout W Steyerberg AU - David M Kent Y1 - 2021/01/01 UR - http://medrxiv.org/content/early/2021/01/25/2021.01.20.21250183.abstract N2 - Objective To assess whether the Prediction model Risk Of Bias ASsessment Tool (PROBAST) and a shorter version of this tool can identify clinical prediction models (CPMs) that perform poorly at external validation.Study Design and Setting We evaluated risk of bias (ROB) on 102 CPMs from the Tufts CPM Registry, comparing PROBAST to a short form consisting of six PROBAST items anticipated to best identify high ROB. We then applied the short form to all CPMs in the Registry with at least 1 validation and assessed the change in discrimination (dAUC) between the derivation and the validation cohorts (n=1,147).Results PROBAST classified 98/102 CPMS as high ROB. The short form identified 96 of these 98 as high ROB (98% sensitivity), with perfect specificity. In the full CPM registry, 529/556 CPMs (95%) were classified as high ROB, 20 (4%) low ROB, and 7 (1%) unclear ROB. Median change in discrimination was significantly smaller in low ROB models (dAUC −0.9%, IQR −6.2%–4.2%) compared to high ROB models (dAUC −11.7%, IQR −33.3%–2.6%; p<0.001).Conclusion High ROB is pervasive among published CPMs. It is associated with poor performance at validation, supporting the application of PROBAST or a shorter version in CPM reviews.What is newHigh risk of bias is pervasive among published clinical prediction modelsHigh risk of bias identified with PROBAST is associated with poorer model performance at validationA subset of questions can distinguish between models with high and low risk of biasCompeting Interest StatementThe authors have declared no competing interest.Clinical TrialN/A not a clinical trialFunding StatementThis research was funded by a Patient-Centered Outcomes Research Institute (PCORI) Methods Award (ME-1606-35555).Author DeclarationsI confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.YesThe details of the IRB/oversight body that provided approval or exemption for the research described are given below:Tufts Health Sciences IRB #12461All necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived.YesI understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).YesI have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.YesWe used publications from the Tufts Predictive Analytics and Comparative Effectiveness (PACE) CPM Registry http://www.pacecpmregistry.org/ ER -