An Assessment of Transparency and Reproducibility-related Research Practices in Otolaryngology

INTRODUCTION

Clinical research serves as the foundation for evidence-based patient care. Given the important role of research in establishing standards of care, reproducible research is critical. However, some authors have expressed concerns that many published research findings may be false or irreproducible.^1–3 In the field of otolaryngology, for example, a study published in JAMA Otolaryngology that investigated the use of dexamethasone for complications post thyroid surgery was recently retracted, partly because of incorrect statistical results that did not support the authors’ conclusions.^4,5 Prior to retraction, the journal requested an investigation into the legitimacy of the authors’ findings. The investigation was never completed because the authors failed to provide the raw data and the original protocol.⁴ Access to raw data sets and protocols—2 components of transparent and reproducible science—is of paramount importance in reducing and correcting false research findings

Overall, only 10% to 25% of biomedical research findings are estimated to be reproducible,^6–9 and scientific research is thus experiencing a “reproducibility crisis.”⁶ Independent verification of results (ie, reproducibility) is key to the scientific process and aids in enhancing the credibility, utility, and reliability of published literature.¹⁰ A survey conducted by the editorial staff of the journal Nature found that more than 70% of the 1500 scientists surveyed were unable to replicate the same outcomes from another scientist’s experiment, and half of these scientists reported failures replicating their own experiments.¹¹ A project dedicated to reproducing key findings in cancer biology experiments was recently abandoned after 32 of 50 replication attempts failed, partly because scientists were unable to obtain sufficient detail to reproduce the original studies.¹² These results suggest significant room exists for improvement with regard to reproducible research.

Irreproducible studies often stem from incomplete reporting of methodology or failing to provide access to materials, protocols, detailed methods, or raw datasets.^13,14 A study comprising 198 publications related to social science reported that only 16% of publications provided access to materials, a few (8%) provided access to raw datasets, none provided access to protocols, and only 3% provided access to analysis scripts.¹⁵ Likewise, a study of biomedical journal publications found that only 1 study provided a full protocol, none made the raw data available, and only 4 were replication studies.¹⁶ Because experimental methodologies are often complex, critically assessing the accuracy of scientific claims and conclusions drawn from them is difficult without access to all relevant study materials.¹⁷ We should question why published scientific research has taken this direction.

The extent of reproducible practices in the field of otolaryngology remains unknown. To date, no studies have explored current practices of the otolaryngology research community with regard to reproducible research practices being included in published results. Here, we evaluated a random sample of the otolaryngology literature using indicators of reproducibility and transparency established by Hardwicke et al.¹⁵ Our goals were to elucidate areas in need of improvement and to establish baseline data for subsequent investigations of the otolaryngology literature.

METHODS

We used the protocol of Hardwicke et al,¹⁵ with modifications explained below, for the present investigation. Because our study did not include human participants, it was not subject to institutional review board oversight per US Department of Health and Human Services’ Code of Federal Regulation 45 CFR 46.102(d) and (f).¹⁸ This study is reported according to the guidance developed by Murad and Wang¹⁹ for meta-research studies and, when relevant, the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA).²⁰ Our protocol, raw data, and other pertinent materials have been deposited on the Open Science Framework (https://osf.io/x24n3/), which has introduced new infrastructure supporting research transparency. Our primary endpoint was an evaluation of indicators of reproducible and transparent research practices within otolaryngology.

RESULTS

Our initial search returned 26,498 records, from which 300 publications were randomly selected for analysis. Of the 300 randomly chosen publications, 286 were included and 14 were excluded either due to inaccessibility or because they had been retracted. Two hundred twenty-four publications had empirical data and 62 did not have empirical data. Furthermore, case reports and case series were excluded from publications with empirical data (n = 151) during select analysis (see Table 1). Figure 1 shows a flow diagram of included and excluded publications, and Table 2 presents the characteristics of the included publications.

• Download figure
• Open in new tab

Figure 1. Flow diagram of included and excluded publications

Discussion

The majority of otolaryngology studies included in our analysis showed substandard procedures relating to reproducibility and transparency. Nearly all the studies lacked a data or material availability statement, did not link to an accessible protocol, and were not preregistered. In addition, most of the studies were not available through open access. Our findings align with previous literature in various fields of medicine.^15,22,23 Many of these irreproducible studies lack specific methodology, resultant data, or access to materials needed to conduct the investigation.^9,24 Irreproducibility carries an expensive price tag; for example, in their retrospective analysis, Freedman et al²⁵ estimated that $28 billion has been spent on preclinical research that is irreproducible. Further, irreproducibility is a contributing factor in the growing public distrust in science.^22,25,26 Complete transparency and reproducibility in research is likely unattainable, but advancement is possible and necessary.

One aspect of reproducible research is the availability of all the data and description of the materials necessary to conduct the experiment. While this study is the first to investigate data availability statements within otolaryngology, lack of data availability and sharing has already been shown to be an issue in other areas of medicine.²⁷ We found that over 94% of analyzed studies did not provide a data or material availability statement. Similarly, an evaluation of 50 high impact factor journals in 2009 revealed that only 9% of studies supplied access to the full data online and 59% did not fully adhere to the data availability policy requirements.²⁸ Such lack of access to raw data and access to materials hinders most efforts at reproducing a study. In response, some journals have begun requiring, or at least encouraging, the inclusion of data availability statements.²⁹ These statements help facilitate replication of the study as well as reanalysis of its outcomes. Given the important role that data availability plays in reproducibility, we recommend that all journals adopt stricter policies on data availability as recently established by the International Committee of Medical Journal Editors (ICMJE) and studied in peer-reviewed journals such as PLOS ONE and BMJ.^27,29,30 PLOS ONE has had a substantial increase over time in studies with data availability statements since enacting the policy.²⁹ Otolaryngology journals, such as Otolaryngology - Head and Neck Surgery and JAMA Otolaryngology - Head and Neck Surgery,³¹ are listed as journals that follow ICMJE guidelines. We recommend all otolaryngology journals consider following suit.

Similar to data and material availability, a preregistration statement was not reported in over 95% of the studies that we analyzed. Preregistration is important because it helps guard against many threats to the integrity of scientific research.^32,33 One specific threat that hinders research quality is selective outcome reporting bias. This bias occurs when some outcomes are reported and some are not, or when a predetermined secondary outcome is presented as the primary outcome.³⁴ Reducing selective outcome reporting bias by preregistration is important in improving research quality overall, but also in improving research reproducibility. Studies become more difficult to reproduce when specific aims and outcomes are not defined a priori.

Another barrier to reproducibility is the lack of open access of published studies, which is especially concerning with regard to publications funded by public tax dollars. Over three-fourths of the studies analyzed in our investigation were not fully accessible, and of the 26 publicly funded studies, only 9 wereavailable for full public access. If a study is financed by public tax dollars, a reasonable expectation is that the public should have unrestricted access to it in total.³⁵ A general increase in open access research across otolaryngology would likely increase reproducibility within the field³⁶ and may also improve the impact factors of otolaryngology journals, which tend to be relatively lower than journals in other areas of medicine.^36,37 Our study corroborated the low impact factors, with a median 5-year impact factor of less than 2 being found for the analyzed otolaryngology journals. Multiple recent initiatives are currently attempting to address transparency, openness, and reproducibility in scientific literature.^{21,33,38–41}

Moving Forward

Given the reproducibility concerns within medical research, substantiated by our findings within otolaryngology, we support recommendations from the scientific literature to improve transparency and reproducibility-related factors (Figure 2). First, we recommend raising awareness of reproducibility issues and supporting behavioral changes through continued independent verification of data. Second, all peer-reviewed otolaryngology journals should consider requiring availability statements regarding materials, data sets, protocols, and analysis scripts (ie, factors related to reproducible research).^42–47 Third, we propose prioritizing innovation of methods that facilitate data sharing compliance for researchers.⁴⁸ Possible solutions include (1) using online repositories, such as Open Data for Science,⁴⁹ Open Science Framework,⁵⁰ or others (see http://www.re3data.org), (2) using OpenTrials,⁵¹ which is designed to share individual patient data from clinical trials in compliance with the Health Insurance Portability and Accountability Act of 1996, and (3) encouraging authors to prioritize preprint servers prior to submitting for publication. Lastly, academic institutions, peer-reviewed journals, and funding providers should consider incentivizing and rewarding investigators who fully disseminate materials, data sets, protocols, and analysis scripts by awarding specific levels or “badges” of reproducibility and transparency as laid out by the Center for Open Science.^52–54 Badges are awarded for open data, open materials, and preregistration. This evidence-based incentive program is free to use for journals and organizations and has dramatically increased the rate of data sharing from 3% to 39% within psychological sciences.^55,56 In our view, implementing these recommendations will support physicians in establishing credible and reliable research that ultimately governs clinical practice, as seen with the landmark Veterans Affairs study on chemoradiation versus surgery and radiation in laryngeal cancer.^57–59 This study is a great example of reproducible research that has stood the test of time and improved patient care.

• Download figure
• Open in new tab

Figure 2. Recommendations for moving forward

Conclusions

Inadequate reproducibility practices exist in otolaryngology which hinders the practice of evidence-based medicine. Research should be conducted and presented in the same way one discusses a complicated case in consultation with a colleague—open and complete access to necessary information, with a genuine and unbiased presentation, to attain the best results.