The epidemiology of systematic review updates: a longitudinal study of updating of Cochrane reviews, 2003 to 2018.

The status of a cohort of Cochrane assessed at three time 2003, 2011, We assessed their subject scope, compiled their publication history using PubMed and CDSR , and compared them to all Cochrane reviews available 2002 and 2017/18. Results: Of the 1,532 Cochrane reviews available in 2002, 11.3% were updated in 2003, with 16.6% not updated between 2003 and 2011. The reviews updated in 2003 were not markedly different to other reviews available in 2002, but more were retracted or declared stable by 2011 (13.3% versus 6.3%). The 2003 update led to a major change of the conclusions of 2.8% of updated reviews (n = 177). The cohort had a median time since publication of the first full version of the review of 18 years and a median of three updates by 2018 (range 1–11). The median time to update was three years (range 0–14 years). By the end of 2018, the median time since the last update was seven years (range 0–15). The median number of included studies rose from eight in the version of the review before the 2003 update, to 10 in that update and 14 in 2018 (range 0–347).


Background
Systematic reviews use explicit formal methods to identify and analyse studies on specific research questions, and synthesise the findings of these studies. (1) Reviews' findings can become outdated if they are overtaken by new studies or data, missing studies or errors in the review or included studies are identified, or methodology improves in critical ways. In a 2007 paper, Shojania et al reported on the need for updates in 100 systematic reviews from 1995 to 2005. (2) They concluded that around half may have been out-of-date, with a signal for required updating within two years of the evidence search in 11% of reviews. The median period without a signal suggesting a need to update was 5.5 years (95% confidence interval (CI), 4.6 to 7.6 years).
The intention to keep reviews up-to-date was a cornerstone of the Cochrane Collaboration. Named for Archie Cochrane, this international network aimed to achieve a goal he articulated by filling the need for: "a critical summary, by specialty or subspecialty, adapted periodically, of all relevant randomized controlled trials".
(3) The Cochrane Collaboration began publishing systematic reviews of the effects of healthcare interventions based on clinical trials in the Cochrane Database of Systematic Reviews (CDSR) in 1995. (4) Its organisers wrote in an introductory brochure in the 1990s that evidence reviews "must be prepared systematically and they must be kept up-to-date to take account of new evidence". (5) A Cochrane review, they wrote, "is updated and amended as new evidence becomes available and errors are identified".
The original expectation was that reviews would be updated at least annually, but in 2000 it was agreed that this was unsustainable, (6) and the expected update interval was changed to every two years, unless a reason was given for a different schedule. (7) However, this also proved to be unsustainable, and in 2016 the organisation moved towards updating by perceived need or priority, preparing a consensus document on updating systematic reviews. (8) With the publication of a revision of the Cochrane Handbook for Systematic Reviews of Interventions (Cochrane Handbook) in 2019, there is no longer any default interval at which Cochrane reviews are judged to be out of date, and the possibility of retracting reviews deemed outdated will cease. (9) Cochrane reviews are the result of a unique large-scale and long-term effort to stay systematically up-to-date with health and social care evidence across a broad spectrum of topics. This collection of reviews also provides an opportunity to study the practice of systematic reviewing and growth of evidence across time.
We assembled a cohort of Cochrane reviews, comprising all the reviews flagged as updated in the CDSR in 2003, and assessed these across 15 years at three time points: 2003, 2011, and 2018. The aims of our study were to describe the cohort of updated reviews in relation to all other Cochrane reviews that were available at the . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. was not certified by peer review) (which The copyright holder for this preprint this version posted December 11, 2019. ; https://doi.org/10.1101/19014134 doi: medRxiv preprint end of 2002 and, in 2011 and around 2018. We also charted and followed up on the updating history of the 2003 update cohort. We assessed how often the 2003 updates led to a major change in reviews' conclusions. In addition, we charted the growth in the number of included studies in the reviews over time. The cohort was established in 2003 by searching each of the four issues of the CDSR published that year for reviews flagged as updated. The issue(s) in which each review was flagged as updated was recorded.

Methods
(a) Subject scope.
Systematic reviews in different subject areas might become outdated more quickly than others. (2) Cochrane reviews are produced and maintained by editorial groups called Cochrane Review Groups (CRGs). These cover one or more subject areas, and have their own editorial policies. Differences in editorial practice could also affect updating. The number of CRGs grew throughout the early years of the Collaboration, with some merging in later years.
In order to gauge how similar in subject and editorial scope the 2003 cohort reviews might be to Cochrane reviews overall, we compared the spread of reviews among CRGS. A list of CRGs in 2019 was collected from the Cochrane website. (10) The names of the CRGs in each year from 1995 to 2002 were collected using the Internet Archive, (11) from a combination of archived issues of Cochrane News (12) and the Cochrane website. (13) The original CRG names were retained, and also normalised to the current CRG name, and mergers of groups were noted.
The CRG base for each of the reviews in the cohort was collected, noting where CRGs which have since been merged into another CRG. We compared the number of CRGs represented in the cohort with the number of CRGs in 2002 and 2019.
To compare the subject mix of the 2003 cohort with current Cochrane review production and maintenance, we gathered the number of new and updated reviews by CRG from the online Cochrane Library, collecting data for both 2017 and 2018 to reduce the impact of annual fluctuations. As the search function does not enable separation of new and updated reviews, the totals of new and updated reviews from 2017 to 2018 were collected from the dashboards on Cochrane's website. (14) (b) Likelihood of being updated.
. CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. was not certified by peer review) (which The copyright holder for this preprint this version posted December 11, 2019. ; https://doi.org /10.1101/19014134 doi: medRxiv preprint In 2011, data was collected to enable comparison of updating history between the 2003 cohort and other Cochrane reviews. In the time since 2003, the CDSR had moved from publication on CD-ROM to online publication, and had a change of publisher. (4,15) It became apparent that many previous versions of Cochrane reviews were no longer published in the CDSR and so we used PubMed to establish a cohort of reviews that had been published in the CDSR by the end of 2002, supplemented by a search of CDSR for reviews still dated pre-2003. The number of reviews that had been published by the end of 2002 was available from another project, (16) but that did not provide identifiers for those reviews.
In 2011, the status of each review that had been available in 2002 was identified using both PubMed and the latest version in the CDSR. There were three possible status designations of Cochrane reviews in 2011 at that time: ongoing, designated stable (no further update required), or withdrawn. Which of these had been applied to each review was recorded. As some of the reviews were no longer included in the CDSR without any withdrawal notice, we assigned that as a fourth category. The CDSR is unusual in having a large number of records withdrawn that could mean either the publication is out of date or that it "contains a major error", and the reason for withdrawal is not routinely stated. (7) Another journal that also withdraws out-of-date publications classifies these all as retracted. (17) Opinions about whether all CDSR withdrawals constitute retractions vary. (18)(19)(20) In this study, the categories withdrawn and no longer included without notice were classified together as retracted.
Both PubMed and CDSR were used to establish whether the reviews had been updated between 2003 and 2011. Dates of updates were defined as the date of publication of a new citation version of the review, and date of search or incorporation of new data if no new cited version was published. When a version of a review involved only a software update, we did not count this as an update.
To compare updating and status between the reviews available in 2002 which had, and had not, been updated in 2003, we analysed the rate of ongoing, stable, and retracted reviews. We also analysed the proportions of ongoing reviews that had not been updated since 2002, those which were updated in 2003 only, and those which had been updated at least once between 2004 and 2011. All reviews flagged as updated in 2003 were collected. To assess the changes to the reviews' conclusions, one author (HB) assessed all of these updates, and another (JD) assessed those published as updates in two of the four issues of the CDSR. Both authors agreed on a final group of reviews that had a major change in their conclusions following the update. 6 The first follow-up of the 2003 cohort of reviews was done in 2011. Using the first issue of CDSR in that year (published in April), the number of years from the review's last reported search for eligible studies to April 2011 was recorded. After publication of the fourth and final issue of the CDSR for 2011, as complete as possible an updating history of the 2003 cohort was assembled. For each review in the cohort, PubMed was searched for previous records of the review. The version of the review published in issue 4 of 2011 was reviewed for information about updates in the three parts of the review that were expected to report the review's history: What's New, history, and notes. In March 2019, we added the year for each update from 2012 to the end of 2018. This was again based on PubMed searches and the information recorded in the reviews' What's New, history, and notes sections. In addition, we added the year the review was first published and the most recent PubMed identifier for the review.
This final data collection for the 2003 cohort alone included an assessment of the status of these reviews at the end of 2018, using the same categories as previously. However, we added a category for 2018: republished after previous retraction. For reviews that were stable, retracted, or republished after previous retraction, we recorded the years of these events.
When we originally identified reviews as updated in 2003, we recorded the number of studies included in that update, as well as the number included in the prior issue of the review. For one review, the updated version of the review was the only one available for assessment. We subsequently recorded the number of included studies in the versions of each review at the end of 2011 and the end of 2018.

Data management and analysis
One author (HB) undertook all data collection, curation, and visualisation. Data for the first collection in 2003 was originally recorded in document tables, and added to an Excel spreadsheet in 2019. All other collections were recorded in Excel. Data was analysed using RStudio 1.1463 running R 3.5.2. (9) Packages tidyverse and (24) reshape2 (25) were used for analyses and data visualisation. Summary statistics were used to describe the cohort. Data for this project, including analytic code, are available at Github. (26) . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. was not certified by peer review) (which The copyright holder for this preprint this version posted December 11, 2019. ; https://doi.org/10.1101/19014134 doi: medRxiv preprint

cohort in perspective: subject scope.
We identified 1,532 Cochrane reviews via PubMed and the CDSR that were available at the end of 2002. This was fewer than the number reported by the Cochrane Collaboration since the beginning of the CDSR in 1995 (1,558). (16) Some reviews had been indexed in PubMed after 2002 (n = 13), and this is likely to be the case for more of the shortfall of 26 reviews. Other reviews may have been retracted prior to the indexing of the CDSR in  Table 1 shows the rise in the number of CRGs between 1995 and 2002, as well as annual review production based on Cochrane-supplied data and the years of first publication of the 173 reviews in the update cohort reviews that had been published by the end of 2002. CRGs had a median of two reviews updated in 2003, ranging from none to 27 (interquartile range (IQR): 3). Only four CRGs had been in existence for less than four years in 2002, and 19 were no more than five years old. At that time annual updating of Cochrane reviews was expected, but only 42 of the 49 CRGs (85.7%) flagged reviews as updated in 2003, and only 11.3% of the reviews published by the end of 2002 were updated during 2003.
To gauge how similar the subject scope of reviews in the 2003 update cohort are to the current subject scope of Cochrane reviews, we compared the spread of reviews among CRGs in the 2003 update cohort with the spread of new and updated reviews published in 2017/18 ( Table 2). The CDSR does not enable breaking these down into new and updated reviews but the 2017/2018 totals reported in Cochrane dashboards online show that 44.8% of these reviews were updates. (14) (The dashboards report fewer total new and updated reviews for 2017/2018: 1,353 versus 1,369 returned by CDSR search).  1,369 reviews). The individual CRGs, based on their 2019 names and grouping, are shown with their proportion of reviews in both time periods in Figure 1.
. CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. was not certified by peer review)   The updating status of the 1,532 reviews published by the end of 2002 was categorised in 2011 to show whether each review was ongoing, stable (no longer being updated), or retracted at that time, and whether or not they had been updated . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. was not certified by peer review) (which The copyright holder for this preprint this version posted December 11, 2019. ; https://doi.org/10.1101/19014134 doi: medRxiv preprint at least once between 2003 and 2011. The retracted reviews included those with a withdrawal notice as well as six reviews that were no longer in the CDSR, with no record to explain their absence. We compared the 173 reviews in the 2003 update cohort for which the original version had been published before 2003 with the other reviews that were available at the end of 2002 (Table 3). . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. was not certified by peer review)   . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. was not certified by peer review) (which The copyright holder for this preprint this version posted December 11, 2019. ; https://doi.org/10.1101/19014134 doi: medRxiv preprint Among the 177 reviews in the 2003 update cohort, 150 were ongoing in 2018: the other 27 were either retracted (n = 15) or designated stable (n = 12). One review had been retracted in 2010, but was updated and republished the following year, and is included among the 150 ongoing reviews.
The median time to each update of these reviews was three years ( Table 4). The shortest was zero in the case of the first update, as some reviews were updated during the year they were published. The longest interval between updates was 14 years between the first and second updates of one review.    . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. was not certified by peer review) (which The copyright holder for this preprint this version posted December 11, 2019. ; https://doi.org/10.1101/19014134 doi: medRxiv preprint Impact of updating over time.
In 2003, the updating of a review resulted in major changes to its conclusions for five reviews (2.8%). We compared the number of included studies in the updated review with the number in its prior version (excluding one review had been retracted without leaving a copy of the original review in CDSR, leading to a single missing baseline value). Some updates did not lead to the inclusion of further studies, but the number of included studies grew substantially over time (median: 8 from the version before the 2003 update to 14 in the version of the review at the end of 2018) ( Figure 5). . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. was not certified by peer review) (which The copyright holder for this preprint this version posted December 11, 2019. ; https://doi.org/10.1101/19014134 doi: medRxiv preprint

Discussion
Our study describes the updating history over 15 years of a cohort of 177 Cochrane reviews that were first updated in 2003. For most of this period, the Cochrane updating policy was a recommended two-year interval until a review was regarded as out of date. By that measure, reviews in this cohort could be considered out of date for more than half of their publication life. However, in another study Shojania et al concluded that the median life of a systematic review before it became outsided was 5.5 years (CI, 4.6 to 7.6 years), based on a sample of 100 systematic reviews. As the median time between updates for the Cochrane reviews in our 2003 update cohort was three years, most of this cohort were likely to be up-to-date most of the time.
However, this cohort has been getting more out of date over time. The median time to first update was three years, but by the end of 2018, more than half the reviews were more than seven years since their last update (range 0-15 years). This is reflective of Cochrane reviews more generally, as more new reviews are published than updated and the legacy of existing reviews now exceeds 8,000. If indeed the interval between review updates continues to increase in the long term, or more and more reviews are never updated, up-to-date Cochrane reviews will become the exception, not the rule.
Until recently, Cochrane policy allowed for withdrawing a review from publication when it was seriously out of date. This resulted in a high retraction rate for our cohort (6.9%), relative to its fellow Cochrane reviews that were not updated in 2003 (4.3%). With the introduction of a new update classification system that aims to phase out the withdrawal of outdated reviews, (28) the retraction rate in future is likely to be considerably lower. The proportion of reviews showing as apparently current in the CDSR but which are seriously outdated will be correspondingly higher.
Our cohort had a higher rate of updates than other Cochrane reviews from the early 2000s. In addition, they included a particularly low rate of "empty" reviews in 2011 (1.1% without included studies) compared to the 8.7% rate Yaffe et al reported for all Cochrane reviews in 2010. (27) In our study, the median number of included studies prior to the 2003 update was eight. Mallett and Clarke reported that the median number of included studies in all Cochrane reviews at the beginning of 2001 was six. (29) The 2019 Cochrane Handbook advises authors of reviews to consider, among other things, whether new eligible studies are likely to be found before deciding to update. (9) Our results are consistent with Cochrane groups already concentrating updating effort on research questions that were generating new studies.
Our results for the earlier years are similar to the results from studies with shorter follow-up than ours. We identified 13 other studies of updating in Cochrane reviews . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. was not certified by peer review) (which The copyright holder for this preprint this version posted December 11, 2019. ; https://doi.org /10.1101/19014134 doi: medRxiv preprint or protocols that addressed some similar outcomes (overview in Appendix). (30)(31)(32)(33)(34)(35)(36)(37)(38)(39)(40)(41)(42) Mean or median times to updates were comparable to those in our study. Our finding that a major change in conclusion after update is rare (2.8%) is also consistent with others' results. Jaidee et al found a 2.0% rate of major changes in their conclusions at the first update of 101 Cochrane reviews. (39) Bashir et al found a 3.9% change in their conclusions in 8 out of 204 reviews with a meta-analysis for a primary outcome. (42) The highest rate was found by French et al, (34) who found a 9.1% change in review conclusions in 23 out of 254 reviews, but these changes were not necessarily major.
There have been two studies with similar analyses of updating non-Cochrane evidence syntheses. The National Institute for Health and Care Excellence (NICE) reported on updating of their systematic review-based clinical practice guidelines. (43) For 11 guidelines, the median time to update was 5.3 years (range: 3.3 to 6.5 years), with major changes in recommendations in six of them. Peterson et al studied 41 comparative effectiveness reviews of drugs, finding a median time to update of just over two years. (44) Our study has several important strengths, particularly its long-term follow-up and open data that could enable others to extend this longitudinal study. The collection of cross-sections of data in 2003, 2011, and 2017/2018 was valuable. By maintaining a historical collection of data and using PubMed as well as the CDSR, we established that the updating and version history of Cochrane reviews in the CDSR has important gaps. While most older versions of reviews are online in the CDSR, some complete reviews and some update versions of reviews have been removed from the CDSR without leaving any public record other than PubMed. Some updated versions of Cochrane reviews were never submitted to PubMed. In some cases this may be in error, but it may also be a matter of policy. (7) This has left no complete public record of all Cochrane reviews, their updates, versions, and retractions. Not submitting updates for indexing at PubMed has critical implications for searches: reviews with misleadingly older dates will not be retrieved by searches that are limited to more recent records.
The inadequate historical record was one of the limitations of our study. We also relied on manual collection of data and quality assurance by a single author. In assessing the growth of included studies, we did not collect data on eligible studies awaiting inclusion and we do not know if this would have affected our results.
Our study raises some issues for Cochrane and others to consider. We could find the date of last search clearly reported in 92.7% of the reviews in 2011. In a study published in 2013, Beller et al found that only 90.0% of a sample of systematic reviews from PubMed reported the date of last search. (45) The availability of that date is critical for users to assess the currency of evidence and the adequacy of overlapping periods in update searches.
. CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. was not certified by peer review) (which The copyright holder for this preprint this version posted December 11, 2019. ; https://doi.org /10.1101/19014134 doi: medRxiv preprint In 2002, Koch drew attention to the poor quality of update reporting in Cochrane reviews, (31) and we found this to be an ongoing problem. Contributing factors include missing versions of Cochrane reviews in the CDSR, and reviews not carrying forward events from "What's New" and other notes sections into the history of each subsequent version of the review as Cochrane advises. (7) Automation of that process could improve this situation.
A further critical gap is the lack of reporting of searches undertaken before it is determined that an update is needed. Although searches that found no new studies were generally reported as updates to the reviews in early years when annual or two-yearly updates were expected, these are no longer as visible. (8,9) Transparency of this element of updating activity is important for users, including other systematic reviewers and producers of clinical practice guidelines and health information generally. A survey of health organisations producing systematic reviews (and often clinical practice guidelines) by Garritty et al in 2010 had 114 respondents (30% of them from Cochrane). Only eight of the organisations did not have any updating activity, but resources for updating for those that did were stretched, and the groups regarded too many of their reviews as too far out of date. Sharing the results of searches is critical in this context to prevent large numbers of groups using resources on the same futile searches.
In 2010, two of us advocated that major changes were needed to keep up with the evidence, given the massive rise in clinical trials and increasing complexity of systematic review methods, with constrained resources. (16) Prioritisation of systematic reviews, reduction of avoidable waste, and "leaner and more efficient methods of staying up-to-date" were stressed. Since then, the rise in clinical trials has escalated: ClinicalTrials.gov amassed over 96,000 trials in its first decade to 2010, and the total nearly doubled in the next five years. (46,47) It now stands at over 320,000 (as of November 2019).
Progress in streamlining methods and avoiding waste are not moving as fast. Methodological expectations for Cochrane reviews have increased, (9,48) and trial registry entries have been added to Cochrane's trial database as the organization grapples with the implications of incorporating unpublished trials and data. (49) There has, however, been a concerted effort to develop and implement methods for prioritisation of updating for Cochrane reviews. (8) With this transition to targeted updating, new questions about processes and impact need to be tackled. Are the right systematic reviews being updated? Are people being harmed by reliance on outdated systematic reviews?
. CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. was not certified by peer review) (which The copyright holder for this preprint this version posted December 11, 2019. ; https://doi.org/10.1101/19014134 doi: medRxiv preprint