Abstract
Background An external control arm is a cohort of control patients that are collected from data external to a single-arm trial. To provide an unbiased estimation of efficacy, the clinical profiles of patients from single and external arms should be aligned, typically using propensity score approaches. There are alternative approaches to infer efficacy based on comparisons between outcomes of single-arm patients and machine-learning predictions of control patient outcomes. These methods include G-computation and Doubly Debiased Machine Learning (DDML) and their evaluation for ECA analysis is insufficient.
Methods We consider both numerical simulations and a trial replication procedure to evaluate the different statistical approaches: propensity score matching, Inverse Probability of Treatment Weighting (IPTW), G-computation, and DDML. The replication study relies on five type 2 diabetes randomized clinical trials granted by the Yale University Open Data Access (YODA) project. From the pool of five trials, observational experiments are artificially built by replacing a control arm from one trial by an arm originating from another trial and containing similarly-treated patients.
Results Among the different statistical approaches, numerical simulations show that DDML has the smallest bias followed by G-computation. Ranking based on mean square error is different with G-computation always being among the lowest-error methods while DDML relative performance improves with increasing sample sizes. For hypothesis testing, DDML controls type-1 error and is conservative whereas G-computation and propensity score approaches can be liberal with type I errors ranging between 5% and 10% in some settings. G-computation is the best method in terms of statistical power, and DDML has comparable power at n = 1000 but its power is inferior to propensity score approaches at n = 250. The replication procedure also indicates that G-computation minimizes mean squared error while DDML has intermediate performances compared to G-computation and propensity score approaches. The confidence intervals of G-computation are the narrowest in lines with its liberal type I error whereas confidence intervals of DDML are the widest that confirms its conservative nature.
Conclusions For external control arm analyses, methods based on outcome prediction models can reduce estimation error and increase statistical power compared to propensity score approaches.
Competing Interest Statement
The authors are employees of Owkin, Inc.
Clinical Trial
ClinicalTrials.gov Identifier: NCT01106625 NCT01137812 NCT01106651 NCT01106677 NCT00968812
Funding Statement
This study did not receive any specific funding
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
The details of the IRB/oversight body that provided approval or exemption for the research described are given below:
A Data User Agreement has been signed between YODA and Owkin that states that approval was received to use the data in the present research work. This study was carried out under YODA Project Protocol #2019-4077. The data that Yale provided consists of Trial data that were provided to Yale by JANSSEN. Trials are controlled, interventional clinical studies in patients of products that have been provided marketing authorization by both the Food and Drug Administration and the European Medicines Agency The 5 trials (NCT01106625, NCT01137812, NTC01106651, NCT01106677, NCT00968812) have all been registered in ClinicalTrials.gov. ClinicalTrials.gov allows the registration of clinical studies with human subjects that assess biomedical and/or health outcomes and that conform to: any applicable human subject or ethics review regulations (or equivalent), any applicable regulations of the national or regional health authority (or equivalent).
I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.
Yes
Data Availability
Data access should be requested to the Yale University Open Data Access (YODA) Project https://yoda.yale.edu/how-request-data
Abbreviations
- ATT
- Average Treatment effect on the Treated
- ATE
- Average Treatment Effect
- C.I.
- Confidence Intervals
- DDML
- Doubly Debiased Machine Learning
- ECA
- External Control Arm Analysis
- IPTW
- Inverse Probability of Treatment Weighting
- MAE
- Mean Absolute Error
- MSE
- Mean Squared Error
- PSM
- Propensity Score Matching
- RCT
- Randomized Clinical Trial
- YODA
- Yale University Open Data Access
