RT Journal Article
SR Electronic
T1 Evidence of unexplained discrepancies between planned and conducted statistical analyses: a review of randomized trials
JF medRxiv
FD Cold Spring Harbor Laboratory Press
SP 2020.02.20.20025684
DO 10.1101/2020.02.20.20025684
A1 Suzie Cro
A1 Gordon Forbes
A1 Nicholas A Johnson
A1 Brennan C Kahan
YR 2020
UL http://medrxiv.org/content/early/2020/02/23/2020.02.20.20025684.abstract
AB Background Choosing or altering the planned statistical analysis approach after examination of trial data (often referred to as ‘p-hacking’) can bias results of randomized trials. However, the extent of this issue in practice is currently unclear. We conducted a review of published randomized trials to evaluate how often a pre-specified analysis approach is publicly available, and how often the planned analysis is changed.Methods A review of randomised trials published between January and April 2018 in six leading general medical journals. For each trial we established whether a pre-specified analysis approach was publicly available in a protocol or statistical analysis plan, and compared this to the trial publication.Results Overall, 89 of 101 eligible trials (88%) had a publicly available pre-specified analysis approach. Only 22/89 trials (25%) had no unexplained discrepancies between the pre-specified and conducted analysis. Fifty-four trials (61%) had one or more unexplained discrepancies, and in 13 trials (15%) it was impossible to ascertain whether any unexplained discrepancies occurred due to incomplete reporting of the statistical methods. Unexplained discrepancies were most common for the analysis model (n=31, 35%) and analysis population (n=28, 31%), followed by the use of covariates (n=23, 26%) and the approach for handling missing data (n=16, 18%). Many protocols or statistical analysis plans were dated after the trial had begun, so earlier discrepancies may have been missed.Conclusions Unexplained discrepancies in the statistical methods of randomized trials are common. Increased transparency is required for proper evaluation of results.Competing Interest StatementThe authors have declared no competing interest.Funding StatementNo specific funding was obtained for this research. Dr Brennan Kahan is grateful for support from the UK Medical Research Council, grant MC_UU_12023/21.Author DeclarationsAll relevant ethical guidelines have been followed; any necessary IRB and/or ethics committee approvals have been obtained and details of the IRB/oversight body are included in the manuscript.YesAll necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived.YesI understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).YesI have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.YesThe datasets used and analysed during the current study are available from the corresponding author on reasonable request.CONSORTConsolidated Standards of Reporting TrialsGEEGeneralized Estimating EquationsIQRInterquartile RangeSAPStatistical Analysis PlanSPIRITStandard Protocol Items: Recommendations for Interventional Trials