An Assessment of Transparency and Reproducibility-related Research Practices in Otolaryngology

Introduction: Clinical research serves as the foundation for evidence-based patient care, and reproducibility of results is consequently critical. We sought to assess the transparency and reproducibility of research studies in otolaryngology by evaluating a random sample of publications in otolaryngology journals between 2014 and 2018. Methods: We used the National Library of Medicine catalog to identify otolaryngology journals that met the inclusion criteria (available in the English language and indexed in MEDLINE). From these journals, we extracted a random sample of 300 publications using a PubMed search for records published between January 1, 2014, and December 31, 2018. Specific indicators of reproducible and transparent research practices were evaluated in a blinded, independent, and duplicate manner using a pilot-tested Google form. Results: Our initial search returned 26,498 records, from which 300 were randomly selected for analysis. Of these 300 records, 286 met inclusion criteria and 14 did not. Among the empirical studies, 2% (95% CI, 0.4%-3.5%) of publications indicated that raw data were available, 0.6% (95% CI, 0.3%-1.6%) reported an analysis script, 5.3% (95% CI, 2.7%-7.8%) were linked to an accessible research protocol, and 3.9% (95% CI, 1.7%-6.1%) were preregistered. None of the publications had a clear statement claiming to replicate, or to be a replication of, another study. Conclusions: Inadequate reproducibility practices exist in otolaryngology. Nearly all studies in our analysis lacked a data or material availability statement, did not link to an accessible protocol, and were not preregistered. Most studies were not available as open access. Taking steps to improve reproducibility would likely also improve patient care.


INTRODUCTION
Clinical research serves as the foundation for evidence-based patient care. Given the important role of research in establishing standards of care, reproducible research is critical. However, some authors have expressed concerns that many published research findings may be false or irreproducible. [1][2][3] In the field of otolaryngology, for example, a study published in JAMA Otolaryngology that investigated the use of dexamethasone for complications post thyroid surgery was recently retracted, partly because of incorrect statistical results that did not support the authors' conclusions. 4,5 Prior to retraction, the journal requested an investigation into the legitimacy of the authors' findings. The investigation was never completed because the authors failed to provide the raw data and the original protocol. 4 Access to raw data sets and protocols-2 components of transparent and reproducible science-is of paramount importance in reducing and correcting false research findings Overall, only 10% to 25% of biomedical research findings are estimated to be reproducible, [6][7][8][9] and scientific research is thus experiencing a "reproducibility crisis." 6 Independent verification of results (ie, reproducibility) is key to the scientific process and aids in enhancing the credibility, utility, and reliability of published literature. 10 A survey conducted by the editorial staff of the journal Nature found that more than 70% of the 1500 scientists surveyed were unable to replicate the same outcomes from another scientist's experiment, and half of these scientists reported failures replicating their own experiments. 11 A project dedicated to reproducing key findings in cancer biology experiments was recently abandoned after 32 of 50 replication attempts failed, partly because scientists were unable to obtain sufficient detail to reproduce the original studies. 12 These results suggest significant room exists for improvement with regard to reproducible research.
Irreproducible studies often stem from incomplete reporting of methodology or failing to provide access to materials, protocols, detailed methods, or raw datasets. 13,14 A study comprising 198 publications related to social science reported that only 16% of publications provided access to materials, a few (8%) provided On May 29, 2019, one of us (DT) conducted a detailed search using the National Library of Medicine catalog for all journals, using the subject term tag otolaryngology [ST]. For inclusion, each journal had to be available in the English language and indexed in MEDLINE. Using this list of journals, an advanced search on PubMed (which catalogues the entire MEDLINE database) was conducted by DT using the electronic or, if unavailable, the linking ISSN numbers of each journal on May 31, 2019. We limited our search to publications between January 1, 2014, and December 31, 2018. We used these search results to obtain a random sample of 300 publications by applying Excel's random number function.

Study Selection
To incorporate a wide range of publications, we did not specify particular study designs for inclusion.
Rather, we included all studies with empirical data for analysis.

Training
Prior to data extraction, 2 of us (AJ, TT) underwent rigorous training to ensure reliability between investigators. The training included an in-person session to review the study design, protocol, extraction form, and location of the information within publications. During training, the 2 investigators extracted data from 2 publications independently; after which, results were compared, and disagreements were reconciled. The investigators then extracted data from 10 additional studies, using the same blinded and double data extraction process. Consensus was achieved by discussion. This training session was recorded and posted online for reference.

Data Extraction
Following training, AJ and TT extracted data from the 300 included publications in a duplicate and blinded fashion. Data extraction was performed between June 3 and 13, 2019, using a pilot-tested Google form. After data extraction was completed, a final consensus meeting was held by the pair to resolve disagreements. A third investigator (DT) was available for adjudication but was not ultimately needed.
. CC-BY 4.0 International license It is made available under a author/funder, who has granted medRxiv a license to display the preprint in perpetuity.
is the (which was not peer-reviewed) The copyright holder for this preprint . https://doi.org /10.1101/19002238 doi: medRxiv preprint During extraction, publications were separated into 2 categories: publications with no empirical data (editorials, commentaries [without reanalysis], simulations, news, reviews, and poems) and publications with empirical data (clinical trial, cohort, case series, case reports, case-control, secondary analysis, chart review, commentary [with data analysis], and cross-sectional). For all empirical and nonempirical publications that we could access, we extracted the funding source, conflict of interest declarations, and impact factor (2016 or 2017 and 5-year impact factors). The variables extracted for empirical publications varied by their study design. For example, we did not evaluate case studies for the presence of a protocol because such studies are unlikely to have a prespecified protocol because they present interesting patient cases as they occur in clinical work. 21 The data extraction protocol for each study design is outlined in Table 1.

Evaluation of Open Access Status
Open access was evaluated because many important components necessary for reproducibility are only available within the full text. Open access status for all 300 publications was evaluated by searching the publication's title or DOI on the website Open Access Button (http://openaccessbutton.org). If publications could not be found on this platform, Google searches (https://www.google.com/) and PubMed searches (https://www.ncbi.nlm.nih.gov/pubmed/) were performed to verify open access status.

Evaluation of Replication and Whether Publications Were Included in Research Synthesis
For empirical studies, we used Web of Science to determine whether the publication was replicated in other studies. For both empirical and nonempirical studies, we used Web of Science to determine the impact factor for each journal and whether the publication had been included in systematic reviews and/or meta-analyses. We performed these tasks by reviewing the title, abstract, and introduction of all publications in which the study was cited. This process was conducted in a duplicate, blinded fashion similar to our data extraction process.
. CC-BY 4.0 International license It is made available under a author/funder, who has granted medRxiv a license to display the preprint in perpetuity.
is the (which was not peer-reviewed) The copyright holder for this preprint . https://doi.org/10.1101/19002238 doi: medRxiv preprint

Data Analysis
Results are presented as frequencies and percentages along with 95% confidence intervals (95% CIs). All analyses were conducted using Microsoft Excel.

RESULTS
Our initial search returned 26,498 records, from which 300 publications were randomly selected for analysis. Of the 300 randomly chosen publications, 286 were included and 14 were excluded either due to inaccessibility or because they had been retracted. Two hundred twenty-four publications had empirical data and 62 did not have empirical data. Furthermore, case reports and case series were excluded from publications with empirical data (n = 151) during select analysis (see Table 1). Figure 1 shows a flow diagram of included and excluded publications, and Table 2 presents the characteristics of the included publications. were cited for analysis by a systematic review or meta-analysis.

Transparent and Reproducible Factors
. CC-BY 4.0 International license It is made available under a author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

Discussion
The majority of otolaryngology studies included in our analysis showed substandard procedures relating to reproducibility and transparency. Nearly all the studies lacked a data or material availability statement, did not link to an accessible protocol, and were not preregistered. In addition, most of the studies were not available through open access. Our findings align with previous literature in various fields of medicine. 15,22,23 Many of these irreproducible studies lack specific methodology, resultant data, or access to materials needed to conduct the investigation. 9,24 Irreproducibility carries an expensive price tag; for example, in their retrospective analysis, Freedman et al 25 estimated that $28 billion has been spent on preclinical research that is irreproducible. Further, irreproducibility is a contributing factor in the growing public distrust in science. 22,25,26 Complete transparency and reproducibility in research is likely unattainable, but advancement is possible and necessary.
One aspect of reproducible research is the availability of all the data and description of the materials necessary to conduct the experiment. While this study is the first to investigate data availability statements within otolaryngology, lack of data availability and sharing has already been shown to be an . CC-BY 4.0 International license It is made available under a author/funder, who has granted medRxiv a license to display the preprint in perpetuity.
is the (which was not peer-reviewed) The copyright holder for this preprint . https://doi.org/10.1101/19002238 doi: medRxiv preprint issue in other areas of medicine. 27 We found that over 94% of analyzed studies did not provide a data or material availability statement. Similarly, an evaluation of 50 high impact factor journals in 2009 revealed that only 9% of studies supplied access to the full data online and 59% did not fully adhere to the data availability policy requirements. 28 Such lack of access to raw data and access to materials hinders most efforts at reproducing a study. In response, some journals have begun requiring, or at least encouraging, the inclusion of data availability statements. 29 These statements help facilitate replication of the study as well as reanalysis of its outcomes. Given the important role that data availability plays in reproducibility, we recommend that all journals adopt stricter policies on data availability as recently established by the International Committee of Medical Journal Editors (ICMJE) and studied in peer-reviewed journals such as PLOS ONE and BMJ. 27,29,30 PLOS ONE has had a substantial increase over time in studies with data availability statements since enacting the policy. 29 Otolaryngology journals, such as Otolaryngology -Head and Neck Surgery and JAMA Otolaryngology -Head and Neck Surgery, 31 are listed as journals that follow ICMJE guidelines. We recommend all otolaryngology journals consider following suit.
Similar to data and material availability, a preregistration statement was not reported in over 95% of the studies that we analyzed. Preregistration is important because it helps guard against many threats to the integrity of scientific research. 32,33 One specific threat that hinders research quality is selective outcome reporting bias. This bias occurs when some outcomes are reported and some are not, or when a predetermined secondary outcome is presented as the primary outcome. 34 Reducing selective outcome reporting bias by preregistration is important in improving research quality overall, but also in improving research reproducibility. Studies become more difficult to reproduce when specific aims and outcomes are not defined a priori.
Another barrier to reproducibility is the lack of open access of published studies, which is especially concerning with regard to publications funded by public tax dollars. Over three-fourths of the studies analyzed in our investigation were not fully accessible, and of the 26 publicly funded studies, only 9 were . CC-BY 4.0 International license It is made available under a author/funder, who has granted medRxiv a license to display the preprint in perpetuity.
is the (which was not peer-reviewed) The copyright holder for this preprint . https://doi.org/10.1101/19002238 doi: medRxiv preprint available for full public access. If a study is financed by public tax dollars, a reasonable expectation is that the public should have unrestricted access to it in total. 35 A general increase in open access research across otolaryngology would likely increase reproducibility within the field 36 and may also improve the impact factors of otolaryngology journals, which tend to be relatively lower than journals in other areas of medicine. 36,37 Our study corroborated the low impact factors, with a median 5-year impact factor of less than 2 being found for the analyzed otolaryngology journals. Multiple recent initiatives are currently attempting to address transparency, openness, and reproducibility in scientific literature. 21,33,38-41

Moving Forward
Given the reproducibility concerns within medical research, substantiated by our findings within otolaryngology, we support recommendations from the scientific literature to improve transparency and reproducibility-related factors (Figure 2). First, we recommend raising awareness of reproducibility issues and supporting behavioral changes through continued independent verification of data. Second, all peerreviewed otolaryngology journals should consider requiring availability statements regarding materials, data sets, protocols, and analysis scripts (ie, factors related to reproducible research). [42][43][44][45][46][47] Third, we propose prioritizing innovation of methods that facilitate data sharing compliance for researchers. 48 Possible solutions include (1) using online repositories, such as Open Data for Science, 49 Open Science Framework, 50 or others (see http://www.re3data.org), (2) using OpenTrials, 51  preregistration. This evidence-based incentive program is free to use for journals and organizations and has dramatically increased the rate of data sharing from 3% to 39% within psychological sciences. 55,56 . CC-BY 4.0 International license It is made available under a author/funder, who has granted medRxiv a license to display the preprint in perpetuity.
is the (which was not peer-reviewed) The copyright holder for this preprint . https://doi.org/10.1101/19002238 doi: medRxiv preprint In our view, implementing these recommendations will support physicians in establishing credible and reliable research that ultimately governs clinical practice, as seen with the landmark Veterans Affairs study on chemoradiation versus surgery and radiation in laryngeal cancer. 57-59 This study is a great example of reproducible research that has stood the test of time and improved patient care.

Strengths and Limitations
Our study has both strengths and limitations. First, a lack of transparency may have been justified for some publications; however, justification was not made explicitly clear within the article. Second, we relied solely on published information for each article. We did not contact any authors because our objective was to measure the reported factors of transparency and reproducibility within published otolaryngology research. Third, the results of this cross-sectional review may not be generalizable to publications in other journals and time periods outside our search. Lastly, we are aware that research that does not employ reproducibility and transparency factors, as assessed in this study, may still add value to the otolaryngology community. . CC-BY 4.0 International license It is made available under a author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

Conclusions
is the (which was not peer-reviewed) The copyright holder for this preprint . https://doi.org/10.1101/19002238 doi: medRxiv preprint CC-BY 4.0 International license It is made available under a author/funder, who has granted medRxiv a license to display the preprint in perpetuity.
is the (which was not peer-reviewed) The copyright holder for this preprint . https://doi.org/10.1101/19002238 doi: medRxiv preprint Tables   Table 1: Measured variables. The variables measured for an individual article varied as it depended on the study design classification. Further details about extraction and coding is available here:https://osf.io/x24n3/

Applicable study design
Role in producing transparent and reproducible science.

Articles
Accessibility and retrieval method (can the article be accessed, is there a public version or is paywall access required?) All (n=300) Open access to scientific research publications may increase visibility of findings, increase collaboration among researchers, and allow for universal access to scientific knowledge. A general increase in open access research would likely increase reproducibility.

Funding
Disclosure statement (are funding sources, or lack of, explicitly declared?) All included studies (n=286) Funding sources can have a strong, influential role on the practices of researchers. Funding sources can also help researchers adjust their experimental design to ensure future funding is not wasted.

Conflict of Interest
Disclosure statement (are conflicts of interest, or lack of, explicitly declared?) All included studies (n=286) Conflict of interest statements allow researchers to disclose pertinent factors or potential influences that may impact study design, conduct, and analysis of results. Full disclosure of any possible conflicts allow for genuine and unbiased presentation of research.

Evidence synthesis
Systematic review and/or metaanalysis citation history (has the article been cited by, and included in the evidence-synthesis component of, a systematic review and/or meta-analysis?) Empirical studies † (n=224) Systematic reviews and meta-analyses incorporate published evidence and evaluate what has previously been done to assess replicated studies.

Protocols
. CC-BY 4.0 International license It is made available under a author/funder, who has granted medRxiv a license to display the preprint in perpetuity.
is the (which was not peer-reviewed) The copyright holder for this preprint  Two to five citations 6 (2.7) † Not accessible through web of knowledge to determine if publication was cited in a systematic review and/or a meta-analysis (www.webofknowledge.com) . CC-BY 4.0 International license It is made available under a author/funder, who has granted medRxiv a license to display the preprint in perpetuity.
is the (which was not peer-reviewed) The copyright holder for this preprint . https://doi.org/10.1101/19002238 doi: medRxiv preprint . CC-BY 4.0 International license It is made available under a author/funder, who has granted medRxiv a license to display the preprint in perpetuity.
is the (which was not peer-reviewed) The copyright holder for this preprint . https://doi.org/10.1101/19002238 doi: medRxiv preprint