ABSTRACT
BACKGROUND Randomized clinical trials (RCTs) define evidence-based medicine, but quantifying their generalizability to real-world patients remains challenging. We propose a multidimensional approach to compare individuals in RCT and electronic health record (EHR) cohorts by quantifying their representativeness and estimating real-world effects based on individualized treatment effects (ITE) observed in RCTs.
METHODS We identified 65 pre-randomization characteristics of an RCT of heart failure with preserved ejection fraction (HFpEF), the Treatment of Preserved Cardiac Function Heart Failure with an Aldosterone Antagonist Trial (TOPCAT), and extracted those features from patients with HFpEF from the EHR within the Yale New Haven Health System. We then assessed the real-world generalizability of TOPCAT by developing a multidimensional machine learning-based phenotypic distance metric between TOPCAT stratified by region including the United States (US) and Eastern Europe (EE) and EHR cohorts. Finally, from the ITE identified in TOPCAT participants, we assessed spironolactone benefit within the EHR cohorts.
RESULTS There were 3,445 patients in TOPCAT and 8,121 patients with HFpEF across 4 hospitals. Across covariates, the EHR patient populations were more similar to each other than the TOPCAT-US participants (median SMD 0.065, IQR 0.011-0.144 vs median SMD 0.186, IQR 0.040-0.479). At the multi-variate level using the phenotypic distance metric, our multidimensional similarity score found a higher generalizability of the TOPCAT-US participants to the EHR cohorts than the TOPCAT-EE participants. By phenotypic distance, a 47% of TOPCAT-US participants were closer to each other than any individual EHR patient. Using a TOPCAT-US-derived model of ITE from spironolactone, all patients were predicted to derive benefit from spironolactone treatment in the EHR cohort, while a TOPCAT-EE-derived model predicted 13% of patients to derive benefit.
CONCLUSIONS This novel multidimensional approach evaluates the real-world representativeness of RCT participants against corresponding patients in the EHR, enabling the evaluation of an RCT’s implication for real-world patients.
Competing Interest Statement
Dr. Thangaraj, Dr. Oikonomou, and Dr. Khera are coinventors of a provisional patent not related to the current work (63/606,203). Dr. Khera is an Associate Editor of JAMA and receives research support, through Yale, from the Blavatnik Foundation, Bristol-Myers Squibb, Novo Nordisk, and BridgeBio. He is a coinventor of U.S. Provisional Patent Applications 63/177,117, 63/428,569, 63/346,610, 63/484,426, 63/508,315, 63/580,137, 63/562,335, and a co-founder of Ensight-AI, Inc and Evidence2Health, LLC. Dr. Oikonomou is an academic co-founder of Evidence2Health LLC, and has been a consultant for Caristo Diagnostics, Ltd and Ensight-AI, Inc. He is a co-inventor in patent applications (US17/720,068, 63/619,241, 63/177,117, 63/580,137, 63/606,203, 63/562,335,WO2018078395A1, WO2020058713A1) and has received royalty fees from technology licensed through the University of Oxford. Dr. Suchard receives grants and contracts from the US Food & Drug Administration, the US Department of Veterans Affairs and Johnson & Johnson, all outside the scope of this work.
Funding Statement
The study is supported by the National Heart, Lung, and Blood Institute of the National Institutes of Health (R01HL167858). Dr. Thangaraj and Dr. Oikonomou are also supported by grants from the National Institutes of Health (5T32HL155000-03 and 1F32HL170592-01, respectively).
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
The details of the IRB/oversight body that provided approval or exemption for the research described are given below:
The Yale Institutional Review Board reviewed this study, and a waiver of consent was granted because it was a retrospective study of medical records.
I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.
Yes
Footnotes
We have updated all sections to include more sensitivity analyses, methodology comparisons, and implications discussions.
Data Availability
The TOPCAT cohort is publicly available through the National Heart, Lung, and Blood Institute Biologic Specimen and Data Repository Information Coordinating Center (BioLINCC) The TOPCAT dataset is available at https://biolincc.nhlbi.nih.gov/studies/topcat/. The Yale electronic health record cohorts are not available due to the use of patient data.
