PT - JOURNAL ARTICLE AU - Dena Zeraatkar AU - Tyler Pitre AU - Gareth Leung AU - Ellen Cusano AU - Arnav Agarwal AU - Faran Khalid AU - Zaira Escamilla AU - Matthew Cooper AU - Maryam Ghadimi AU - Ying Wang AU - Francisca Verdugo AU - Gabriel Rada AU - Elena Kum AU - Anila Qasim AU - Jessica J Bartoszko AU - Reed Siemieniuk AU - Chirag J. Patel AU - Gordon Guyatt AU - Romina Brignardello-Petersen TI - The trustworthiness and impact of trial preprints for COVID-19 decision-making: A methodological study AID - 10.1101/2022.04.04.22273372 DP - 2022 Jan 01 TA - medRxiv PG - 2022.04.04.22273372 4099 - http://medrxiv.org/content/early/2022/04/04/2022.04.04.22273372.short 4100 - http://medrxiv.org/content/early/2022/04/04/2022.04.04.22273372.full AB - Purpose To assess the trustworthiness and impact of preprint trial reports during the COVID-19 pandemic.Data sources WHO COVID-19 database and the L-OVE COVID-19 platform by the Epistemonikos Foundation (up to August 3rd, 2021)Design We compare the characteristics of COVID-19 trials with and without preprints, estimate time to publication of COVID-19 preprint reports, describe discrepancies in key methods and results between preprint and published trial reports, report the number of retracted preprints and publications, and assess whether including versus excluding preprint reports affects meta-analytic estimates and the certainty of evidence. For the effects of eight therapies on mortality and mechanical ventilation, we performed meta-analyses including preprints and excluding preprints at 1 month, 3 months, and 6 months after the first trial addressing the therapy became available either as a preprint or publication (120 meta-analyses in total).Results We included 356 trials, 101 of which are only available as preprints, 181 as journal publications, and 74 as preprints first and subsequently published in journals. Half of all preprints remain unpublished at six months and a third at one year. There were few important differences in key methods and results between trial preprints and their subsequent published reports. We identified four retracted trials, three of which were published in peer-reviewed journals. With two exceptions (2/60; 3.3%), point estimates were consistent between meta-analyses including versus excluding preprints as to whether they indicated benefit, no appreciable effect, or harm. There were nine comparisons (9/60; 15%) for which the rating of the certainty of evidence differed when preprints were included versus excluded, for four of these comparisons the certainty of evidence including preprints was higher and for five of these comparisons the certainty of evidence including preprints was lower.Limitations The generalizability of our results is limited to COVID-19. Preprints that are subsequently published in journals may be the most rigorous and may not represent all trial preprints.Conclusion We found no compelling evidence that preprints provide less trustworthy results than published papers. We show that preprints remain the only source of findings of many trials for several months, a length of time that is unacceptable in a health emergency. We show that including preprints may affect the results of meta-analyses and the certainty of evidence. We encourage evidence users to consider data from preprints in contexts in which decisions are being made rapidly and evidence is being produced faster than can be peer-reviewed.Summary Box 1What is already known on this topicWhat is already known on this topicClinicians and decision-makers need rapidly available and credible data addressing the comparative effectiveness of treatments and prophylaxis for COVID-19.Investigators have adopted preprint servers, which allow the rapid dissemination of research findings before publication in peer-reviewed journals.What this study addsWhat this study addsWe found no compelling evidence that preprints provide less trustworthy results than published papers.We show that including preprints may affect the results of meta-analyses and the certainty of evidence and we encourage evidence users to consider data from preprints in contexts in which decisions are being made rapidly and evidence is being produced faster than can be peer-reviewed.Competing Interest StatementThe authors have declared no competing interest.Funding StatementThis study was supported by the Canadian Institutes of Health Research CIHR-IRSC:057900132 and Coronavirus Rapid Research Funding Opportunity - OV2170359. DZ is funded by a Banting Postdoctoral FellowshipAuthor DeclarationsI confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.YesI confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.YesI understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).YesI have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.YesUpon a reasonable request to the corresponding author. https://osf.io/9adxb/.