Abstract
Background Choosing or altering the planned statistical analysis approach after examination of trial data (often referred to as ‘p-hacking’) can bias results of randomized trials. However, the extent of this issue in practice is currently unclear. We conducted a review of published randomized trials to evaluate how often a pre-specified analysis approach is publicly available, and how often the planned analysis is changed.
Methods A review of randomised trials published between January and April 2018 in six leading general medical journals. For each trial we established whether a pre-specified analysis approach was publicly available in a protocol or statistical analysis plan, and compared this to the trial publication.
Results Overall, 89 of 101 eligible trials (88%) had a publicly available pre-specified analysis approach. Only 22/89 trials (25%) had no unexplained discrepancies between the pre-specified and conducted analysis. Fifty-four trials (61%) had one or more unexplained discrepancies, and in 13 trials (15%) it was impossible to ascertain whether any unexplained discrepancies occurred due to incomplete reporting of the statistical methods. Unexplained discrepancies were most common for the analysis model (n=31, 35%) and analysis population (n=28, 31%), followed by the use of covariates (n=23, 26%) and the approach for handling missing data (n=16, 18%). Many protocols or statistical analysis plans were dated after the trial had begun, so earlier discrepancies may have been missed.
Conclusions Unexplained discrepancies in the statistical methods of randomized trials are common. Increased transparency is required for proper evaluation of results.
Competing Interest Statement
The authors have declared no competing interest.
Funding Statement
No specific funding was obtained for this research. Dr Brennan Kahan is grateful for support from the UK Medical Research Council, grant MC_UU_12023/21.
Author Declarations
All relevant ethical guidelines have been followed; any necessary IRB and/or ethics committee approvals have been obtained and details of the IRB/oversight body are included in the manuscript.
Yes
All necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.
Yes
Data Availability
The datasets used and analysed during the current study are available from the corresponding author on reasonable request.
List of abbreviations
- CONSORT
- Consolidated Standards of Reporting Trials
- GEE
- Generalized Estimating Equations
- IQR
- Interquartile Range
- SAP
- Statistical Analysis Plan
- SPIRIT
- Standard Protocol Items: Recommendations for Interventional Trials
