Results publications are inadequately linked to trial registrations: An automated pipeline and evaluation of German university medical centers

1 Introduction

Linking trial registrations and results publications makes each more findable and improves research transparency at minimal effort to researchers. Threaded evidence also empowers readers to cross-check sources, e.g., for potential outcome switching (Altman, Furberg, Grimshaw, & Shanahan, 2014). If trial results cannot be found or linked to their registration, systematic reviewers may miss relevant data and draw invalid conclusions (Miron, Gonçalves, & Musen, 2020). Health policy decisions and clinical guidelines rely on such evidence synthesis, and incomplete evidence can misinform subsequent clinical trials, drive the misallocation of healthcare resources, and risk patient wellbeing (Chan et al., 2014; Council for International Organizations of Medical Sciences & World Health Organization, 2016).

To inform clinical decision making with comprehensive trial evidence, legal regulations and ethical guidelines advocate results transparency (Altman & Moher, 2014; Borysowski, Wnukiewicz-Kozłowska, & Górski, 2020; Chalmers, Glasziou, & Godlee, 2013; World Health Organization, 2017; World Medical Association, 2013). Trial results should be bidirectionally linked, meaning both a reference to publications in the registration, and trial registration numbers (TRNs) in the full-text, abstract, and metadata of publications. The Consolidated Standards of Reporting Trials (CONSORT) as well as the International Committee of Medical Journal Editors (ICMJE) ask trialists to report the “trial registration number and name of the trial register” in both the full-text and abstract of trial results publications (Hopewell, Clarke, Moher, Wager, Middleton, Altman, Schulz, & Group, 2008; International Committee of Medical Journal Editors (ICMJE), 2019). Reporting a TRN solely in the full-text does not suffice, since readers may not have access to the full-text or may judge a trial on the abstract alone (Hopewell, Clarke, Moher, Wager, Middleton, Altman, & Schulz, 2008). Abstracts may also be published independently of a full-text publication, such as for conferences (Hopewell, Clarke, Moher, Wager, Middleton, Altman, Schulz, & Group, 2008).

The inclusion of TRNs in publication and bibliographic metadata especially enhances machine readability and discoverability. Trial results can then be found more efficiently using TRNs to search publication databases (Huser & Cimino, 2013b). Publications can then also be automatically linked within trial registrations, as is currently done with ClinicalTrials.gov using TRN metadata from PubMed (National Library of Medicine, 2017). Such references to results publications within a registration allow readers to quickly identify and navigate to the trial findings. Results references may also be entered manually in selected registries, with varying degrees of structure (Miron et al., 2020; Moja et al., 2009; Venugopal & Saberwal, 2021). ClinicalTrials.gov, for example, provides fields for DOI and PubMed identifier (PMID), whereas the German Clinical Trials Registry (DRKS) offers an unstructured free-text field. While registry readers may be able to find the referenced publication from the free-text, unique identifiers provide a more robust and structured link that allows for automated retrieval (Huser & Cimino, 2013b).

Trial evidence appears to be insufficiently linked in both registrations and publications. Previous studies on trial registrations found links to results publications in as few as 13% to as many as 60% of trials (Bashir, Bourgeois, & Dunn, 2017; Huser & Cimino, 2013b; Wetering, Scholten, Haring, Clarke, & Hooft, 2012). TRN reporting in either the metadata, abstract, or full-text of trial publications has been found to be as low at 8% and as high as 97%, varying widely depending on how the trial population was defined (Al-Durra, Nolan, Seto, & Cafazzo, 2020; Bashir et al., 2017; Carlisle, 2020; Huser & Cimino, 2013a; Wetering et al., 2012). The sampling strategies used in previous studies limit our ability to draw conclusions about registration-publication linking. Publication-based cohorts relied on inaccurate PubMed clinical trial filters (Glanville, Kotas, Featherstone, & Dooley, 2020) and included publications beyond trial results, as well as trials with unknown registration rates, as not all trials are registered, (Denneny, Bourne, & Kolstoe, 2019; Gopal et al., 2018; Trinquart, Dunn, & Bourgeois, 2018). Registration-based cohorts included trials that may not have an associated publication and thus should not be expected to have a publication link (Chen et al., 2016; Dwan, Gamble, Williamson, & Kirkham, 2013; Ross et al., 2012). Furthermore, these studies of publication links in registrations have primarily focused on ClinicalTrials.gov, and we are not aware of previous studies that investigate publication links in DRKS. In this study, we limit our sample to registered clinical trials with published results which can be expected to link the registration and publication, and which allows us to draw conclusions about the prevalence of this responsible research practice.

2 Methods

3 Results

The IntoValue dataset includes all trials led by a German UMC that were registered on ClinicalTrials.gov or DRKS and completed between 2009 and 2017 (n = 3790). After applying our exclusion criteria, our sample included 1895 trials with 1861 unique results publications indexed in PubMed and available as full-text, as some publications include results from more than one trial. Appendix A provides a flow diagram of the trial and publication screening. Table 1 shows summary descriptive details of the trials with results publications by registry.

Per our inclusion criteria all trials in our sample (n = 1895) were registered and had a publication, however only 373 (20%) trials had the most comprehensive registration-publication linking, meaning the publication linked in the registration as well as the TRN in the publication full-text, abstract, and PubMed metadata. Disregarding metadata, which is largely beyond trialists’ control, an additional 12 trials had comprehensive linking for a total of 385 (20%). An additional 92 (4.9%) met the CONSORT and ICMJE guidelines to include TRNs in both the full-text and the abstract. In contrast, we found 327 (17%) trials with no links in either the registration or the publication. The most common linking practice was reporting of the TRN in the full-text only, accounting for 476 (25%) trials. The inclusion of the various link types ranged from 715 (38%) in abstracts to 1,137 (60%) in full-text. Table 2 shows registration-publication links overall and by registry. Figure 1 shows the intersection of registration-publication links across the various types for ClinicalTrial.gov and DRKS trials.

• Download figure
• Open in new tab

Figure 1. Trials with all combinations of links between registration and publication in ClinicalTrials.gov and DRKS

Table 3 shows the crude univariate and adjusted multivariate odds ratios (ORs) for each type of publication-registration link across all explanatory variables. Completion year did not have a strong relationship with linking practices, although trials completed more recently were more likely to report the TRN in the abstract (aOR 1.13 [1.07, 1.19]) and full-text (aOR 1.07 [1.03, 1.12]). Figure 2 shows the rate of registration-publication links over time for ClinicalTrials.gov and DRKS.

• Download figure
• Open in new tab

Figure 2. Percentage of trials with linked registrations and publications by trial completion year in ClinicalTrials.gov and DRKS.

Completion year from registry.

Across the adjusted multivariate models, trials with one link type were generally more likely to have another type of link. In particular, TRNs were more likely to appear in the PubMed metadata if included in the abstract (aOR 24.2 [16.2, 37.1]), but not more likely if included in the full-text (aOR 1.33 [0.95, 1.87]). DRKS TRNs were less likely than ClinicalTrials.gov TRNs to appear in the PubMed metadata (aOR 0.22 [0.14, 0.33]). Similarly, for trials with a TRN in either abstract or full-text, ClinicalTrials.gov TRNs appeared at a higher rate than DRKS TRNs in the metadata (62% vs. 18%). Appendix B shows the number and proportion of trials with a TRN in the metadata given a TRN in the abstract, full-text, or either, both overall and by registry.

Trials registered in DRKS were also less likely than trials registered in ClinicalTrials.gov to reference the primary outcome publication in the registration (aOR 0.39 [0.27, 0.55]). Most (78%, 662/849) ClinicalTrials.gov trials that reference the publication in the registration had the link automatically derived from the PubMed metadata, which is not currently possible for DRKS. Excluding trials with automatically linked publications, ClinicalTrials.gov and DRKS had similar rates of referencing publication in the registration (24%, 187/786 vs. 22%, 97/447). Trials also had additional publications linked in the registrations, up to 72 in ClinicalTrials.gov (NCT00646412) and 13 in DRKS (DRKS00007770).

In our manual evaluation of the TRN regular expressions for classifying PubMed secondary identifiers (n = 1296), we found that most ids were clear true positive TRNs (n = 1288). A handful (n = 3) were clear true negative other ids, meaning ids from non-registry databanks, such as molecular sequences and open data repositories (i.e., figshare, Dryad) (National Library of Medicine, 2018). We also found 5 ids which we manually classified as true negative non-TRNs; however, using additional PubMed data, we determined these were severely misformatted DRKS ids (i.e., missing the preceding letters “DRKS” and just a string of numbers such as “00000711”). Our regexes correctly classified all well-formatted TRNs and non-TRNs, resulting in a sensitivity and specificity of 100%. If we had instead categorized the 5 severely misformatted DRKS ids as true positives, our regexes would have sensitivity of 99.6% and a specificity of 100%.

4 Discussion

Linking of trial registrations and results publications plays an important role in research transparency and facilitates comprehensive evidence synthesis and informed health policy decision making. Poor linking poses a barrier to identifying trial publications via automated approaches and instead requires researchers to perform intensive manual searches to attempt to match publications to trials (Bashir et al., 2017; Chen et al., 2016; Wieschowski et al., 2019). This responsible research practice comes at minimal costs to researchers, from seconds for pasting TRNs in papers, to minutes for adding a publication link to the registration.

Our study shows that German UMCs can improve in both TRN reporting in publications and references to publications in the registration. In our sample (n = 1895), 17% (327) of trials had no links between registration and publication and only 20% (373) of trials had the most comprehensive registration-publication links. Furthermore, only 25% (477) of trials in our sample fully met the CONSORT and ICMJE guidelines to include TRNs in both the full-text and the abstract. Linking practices showed at best minimal improvement over time. The upward trend in reporting in full-text and abstracts suggests this practice is gaining traction, however more slowly than advisable per CONSORT and ICMJE guidelines. Trials registered in ClinicalTrials.gov were overall better linked than trials registered in DRKS. These differences are in part beyond trialists’ control and reliant on bibliometric databases (i.e., PubMed) and registries, namely (1) generating PubMed metadata from TRNs in the abstract or full-text, and (2) automated indexing of publications in the registry.

Our findings suggest that PubMed’s current approach to capturing TRNs in metadata misses TRNs from the full-text as well as DRKS trials. TRNs in PubMed metadata may be either provided by publishers or manually assigned by National Library of Medicine (NLM) staff who copy-and-paste TRNs found in the abstract and full-text (personal communication via NLM Helpdesk Ticket CAS-552810-T3H7V5). As such, we expect a TRN from any registry to be included in the metadata, if the trialist includes it in either the abstract or the full-text. However, we found that while trials with a TRN in the abstract were indeed more likely to include the TRN in the metadata (aOR 24.2 [16.2, 37.1]), TRNs in the full-text were not more likely to appear in the metadata (aOR 1.33 [0.95, 1.87]). Furthermore, our data suggest that NLM staff are better at extracting ClinicalTrials.gov than DRKS TRNs (aOR 0.22 [0.14, 0.33]).

Automated indexing of publications in ClinicalTrials.gov accounts for most (78%, 662/849) publication references in the registry and drives the almost three-fold discrepancy with DRKS (59%, 849/1,448 vs. 22%, 97/447). Currently, DRKS allows for only manual submission of references by trialists and does not index publications. Furthermore, registrations may refer to additional publications beyond results (e.g., up to 72 publications for one ClinicalTrials.gov registration in our sample), so registry metadata should encode publication type to support quick identification of results. While such publication type metadata is currently available in both registries, it is not systematically used, and many publications are categorized generically as a “paper” (DRKS) or “reference” (ClinicalTrials.gov).

Accurate TRN formatting in publication data is critical for machine-readability, which in turn enables automated indexing of publications in the registration. While our regular expressions allow for and correct minor formatting errors (such as erroneous punctuation), egregious misformatting may make the TRN undetectable. For example, in our visual inspection of the PubMed metadata TRNs, we found 5 severely misformatted DRKS TRNS (i.e., numbers only with no preceding letters), which we could only identify as DRKS TRNs using additional metadata and which prevented the regex from classifying them as TRNs based on the pattern alone.

6 Funding

This work was funded under a grant from the Federal Ministry of Education and Research of Germany (Bundesministerium für Bildung und Forschung - BMBF) [01PW18012]. The funder had no involvement in the study design, data collection, analysis, or interpretation, writing of the manuscript, or the decision to submit for publication.

7 Author disclosures

MSH, BGC, and DS declare they have no direct conflict of interest related to this work. MSH is employed as a researcher under the project funding and additional grants from the German Bundesministerium für Bildung und Forschung (BMBF). DS is a member of the Sanofi Advisory Bioethics Committee and receives an honorarium for his contribution to meetings.