ABSTRACT
OBJECTIVE Before integrating new machine learning (ML) into clinical practice, algorithms must undergo validation. Validation studies require sample size estimates. Unlike hypothesis testing studies seeking a p-value, the goal of validating predictive models is obtaining estimates of model performance. Our aim was to provide a standardized, data distribution- and model-agnostic approach to sample size calculations for validation studies of predictive ML models.
MATERIALS AND METHODS Sample Size Analysis for Machine Learning (SSAML) was tested in three previously published models: brain age to predict mortality (Cox Proportional Hazard), COVID hospitalization risk prediction (ordinal regression), and seizure risk forecasting (deep learning). The SSAML steps are: 1) Specify performance metrics for model discrimination and calibration. For discrimination, we use area under the receiver operating curve (AUC) for classification and Harrell’s C-statistic for survival models. For calibration, we employ calibration slope and calibration-in-the-large. 2) Specify the required precision and accuracy (≤0.5 normalized confidence interval width and ±5% accuracy). 3) Specify the required coverage probability (95%). 4) For increasing sample sizes, calculate the expected precision and bias that is achievable. 5) Choose the minimum sample size that meets all requirements.
RESULTS Minimum sample sizes were obtained in each dataset using standardized criteria.
DISCUSSION SSAML provides a formal expectation of precision and accuracy at a desired confidence level.
CONCLUSION SSAML is open-source and agnostics to data type and ML model. It can be used for clinical validation studies of ML models.
Competing Interest Statement
The authors have declared no competing interest.
Funding Statement
Dr. Goldenholz received funding from NIH 5KL2TR002542. Dr. Westover was supported by the Glenn Foundation for Medical Research and American Federation for Aging Research (Breakthroughs in Gerontology Grant); American Academy of Sleep Medicine (AASM Foundation Strategic Research Award); Football Players Health Study (FPHS) at Harvard University; Department of Defense through a subcontract from Moberg ICU Solutions, Inc; and NIH (R01NS102190, R01NS102574, R01NS107291, RF1AG064312, R01AG062989). MBW is a co-founder of Beacon Biosignals.
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
The details of the IRB/oversight body that provided approval or exemption for the research described are given below:
3 Studies were used for this paper. The ST study was determined exempted by the Beth Israel Deaconess Medical Center Institutional Review Board, 2017D000488. The COVA and BAI studies were determined exempted by the Mass General Brigham Institutional Review Board, 2013P001024
I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.
Yes
Footnotes
daniel.goldenholz{at}bidmc.harvard.edu, hsun8{at}mgh.harvard.edu, WGANGLBERGER{at}mgh.harvard.edu, mwestover{at}mgh.harvard.edu
CONFLICT OF INTEREST DISCLOSURES: DG, HS, WG and MBW have no relevant conflict of interests to disclose.
ROLE OF FUNDER/SPONSOR: Dr. Goldenholz received funding from NIH 5KL2TR002542. Dr. Westover was supported by the Glenn Foundation for Medical Research and American Federation for Aging Research (Breakthroughs in Gerontology Grant); American Academy of Sleep Medicine (AASM Foundation Strategic Research Award); Football Players Health Study (FPHS) at Harvard University; Department of Defense through a subcontract from Moberg ICU Solutions, Inc; and NIH (R01NS102190, R01NS102574, R01NS107291, RF1AG064312, R01AG062989). MBW is a co-founder of Beacon Biosignals.
Data Availability
All data produced in the present study are available upon reasonable request to the authors
