Curating and translating the evidence about SARS-CoV-2 and COVID-19 for frontline public health and clinical care: The Novel Coronavirus Research Compendium (NCRC)

The public health crisis created by the SARS-CoV-2 pandemic has spurred a deluge of scientific research aimed at informing public health and medical response to the COVID-19 pandemic. However, those working in frontline public health and clinical care had insufficient time to parse the rapidly evolving evidence and use it for decision making. Academics in public health and medicine were well-placed to translate the evidence for use by frontline clinicians and public health practitioners. The Novel Coronavirus Research Compendium (NCRC), a group of >50 faculty and trainees, began in March 2020 with the goal to quickly triage and review the large volume of preprints and peer-reviewed publications on SARS-CoV-2 and COVID-19, and to summarize the most important, novel evidence to inform pandemic response. From April 6, 2020 through January 1, 2021, 54,192 papers and preprints were screened by NCRC teams and 527 were selected for review and uploaded to the NCRC website for public consumption. The majority of papers reviewed were peer-reviewed publications (n=395, 75%), published in 102 journals; 25% (n=132) of papers reviewed were of preprints. The NCRC is a successful model of how academics can support practitioners by translating scientific knowledge into action and help to build capacity among students for this work. This approach could be used for health problems beyond COVID-19, but the effort is resource intensive and may not be sustainable over the long term.


Introduction 37
The public health crisis created by the SARS-CoV-2 pandemic has spurred an unprecedented 38 response from the public health and scientific community to generate evidence about 39 transmission, clinical presentation, pathogenesis, and best practices for prevention and 40 mitigation. Between January 30 and April 23, 2020, there were an average of 367 articles 41 published weekly about SARS-CoV-2 with a median submission-to-publication time of six days 42 (Palayew et al. 2020). In comparison, when the World Health Organization declared Ebola as an 43 international emergency in 2019, there were four articles published weekly with a median 44 submission-to-publication time of 15 days. More than 100,000 papers were published on SARS-45 CoV-2 between December 2019 and December 2020 (https://coronacentral.ai/), more than all 46 papers ever published for infectious diseases such as measles (~53,000) or Lyme disease 47 (~24,000) 1 . The pandemic emergency has necessitated such urgent sharing of scientific 48 evidence that many authors have increasingly turned to preprints to share results quickly 49 despite the acceleration in publication speed (Horbach 2020). 50 The large volume of new evidence about SARS-CoV-2 was produced with the aim of improving 51 our collective medical and public health response; however, this aspiration can only be realized 52 if, at a minimum, the best evidence is seen by practitioners and policy makers at the right time. 53 Early in the pandemic, two barriers to optimal use of emerging evidence became clear. to have the required technical expertise to appropriately evaluate the evidence across broad 59 topic areas and to determine its relevance to clinical or public health practice. It can be even 60 more difficult to assess the strengths and weaknesses of evidence presented in preprints 61 because they, by definition, are often less developed than published papers. 62 Academics in public health and medicine are well-placed to help facilitate the use of this 63 evidence by frontline clinicians and public health practitioners. They routinely review and critique 64 the scientific literature, and collectively have the technical training to assess strengths and 65 weaknesses of studies across a wide variety of scientific fields. 66

67
Purpose 68 The Novel Coronavirus Research Compendium (NCRC) began in March 2020, with the goal to 69 quickly triage and review the large volume of preprints and peer-reviewed publications on 70 SARS-CoV-2 and COVID-19, and to summarize the most important, novel evidence to inform 71 health departments, clinicians, and policymakers responsible for pandemic response. Here, we 72 present our process and experiences with this initiative to provide a case study in knowledge 73 translation for pandemic response and to serve as a reference for others considering similar 74 evidence curation efforts for other diseases. 75

Composition and expertise of teams 77
The NCRC comprises eight teams focused on clinical presentation of COVID-19, diagnostics, 78 ecology and spillover, epidemiology, disease modeling, non-pharmaceutical interventions, 79 pharmaceutical interventions, and vaccines. Each topic is led by faculty with expertise in that 80 area, supported by other faculty, doctoral trainees, postdoctoral fellows, or medical students. 81 for use under a CC0 license. This article is a US Government work. It is not subject to copyright under 17 USC 105 and is also made available (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. project.org/web/packages/rvest/index.html) to extract relevant information. For preprints from 92 bioRxiv and medRxiv, papers are obtained from their "COVID-19 SARS-CoV-2 preprints" 93 curated collection (https://connect.biorxiv.org/relate/content/181) using their application 94 programming interface (API) in the JSON format using the jsonlite R package (https://cran.r-95 project.org/web/packages/rvest/index.html). Preprints from SSRN are obtained as daily XML 96 files through the Elsevier Developers API and are processed as above. All preprints are 97 assigned to each of the research areas using predefined search queries. Articles that match 98 multiple research areas are assigned to the group with the fewest papers. 99 Metadata about papers returned by this process are appended to a Google Sheets spreadsheet 100 that serves as the back-end database for an R Shiny web application, which was developed to 101 be the primary interface for NCRC teams to access and select the papers generated through 102 the automated search results. 103 In the "triage" process, NCRC members selected papers for in-depth review that (1) contain 104 original research (can include systematic reviews) representing important contributions to our 105 for use under a CC0 license.
This article is a US Government work. It is not subject to copyright under 17 USC 105 and is also made available (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.
The copyright holder for this preprint this version posted April 27, 2021. ; https://doi.org/10.1101/2021.04.26.21255437 doi: medRxiv preprint understanding of the pandemic that would be relevant to a practice-based public health 106 audience and/or (2) are widely circulated in the public sphere. Studies are also sometimes 107 identified by members through news reports or social media before the automated process; 108 these are processed outside of the R Shiny application. 109

Structure of the reviews and editing 110
NCRC members read articles selected for review and identified the study population and 111 design, highlighted the major findings, study strengths and limitations, and summarized the 112 added value of the evidence considering what was already known. Each review also included a 113 section called 'Our Take,' which provides a capsule evaluation in ~150 words. Reviews are 114 typically drafted by doctoral trainees and reviewed by faculty before being entered into the R 115 Shiny application. These submitted reviews undergo a second round of faculty editing for clarity 116 and consistency of communication. Final reviews are then automatically posted to the NCRC 117 website (https://ncrc.jhsph.edu/) directly from a Google Sheet, which automatically populates 118 sections. Through a formal collaboration with bioRxiv and medRxiv, NCRC reviews of most 119 preprint papers are also automatically posted onto the preprint's bioRxiv or medRxiv page. 120

Communicating reviews 121
The NCRC website launched on April 27, 2020. Paper reviews were organized across the eight 122 research areas (with the ability to cross-post to all relevant areas). A search function allows 123 viewers to search for reviews based on words appearing anywhere in the review. On June 15, 124 2020, each review began to include the date it was published to the NCRC website. Two 125 reviews were of papers eventually retracted due to lack of data reproducibility. For these 126 papers, the phrase "This article has been retracted due to concerns over data veracity" replaced 127 our reviews on the website. 128 for use under a CC0 license.
This article is a US Government work. It is not subject to copyright under 17 USC 105 and is also made available (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. On July 2, 2020, an email newsletter was launched where subscribers received a weekly digest 129 of all new reviews posted to the website. The team has also used Twitter (@JHSPH_NCRC) to 130 highlight noteworthy articles and to post topical threads with an evidence summary and links to 131 multiple related reviews. Popular hashtags (e.g., #COVID19, #SARSCoV2, and #coronavirus) 132 are used to help people who are not yet following @JHSPH_NCRC to find these posts. 133 Outcomes 134 From April 6, 2020 through January 1, 2021, 54,192 papers and preprints were uploaded into 135 the R Shiny application for teams to triage and review. The number of articles uploaded to Shiny 136 increased through August and decreased in the following months (Table 1). The majority of the 137 papers uploaded to Shiny were publications in peer-reviewed journals (76%), while the other 138 24% were from preprint archives (bioRxiv, medRxiv, SSRN, and Research Square) (Table 1). 139 Teams posted 527 total reviews to the NCRC website as of January 1, 2021; 395 (75%) were of 140 peer-reviewed publications and 132 (25%) were of preprints (Table 1, Figure 1 This article is a US Government work. It is not subject to copyright under 17 USC 105 and is also made available (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The NCRC has been covered by media outlets, including Buzzfeed, Vox, and The New York 160 Times, and scientific magazines such as Science. NCRC faculty have also appeared on the 161 "Public Health on Call" podcast produced by the Johns Hopkins Bloomberg School of Public 162 Health to discuss recent reviews. 163 164

Lessons learned 165
The NCRC represents a large, coordinated, multi-disciplinary effort of technical experts in public 166 health science and clinical medicine dedicated to using their skills to support frontline clinicians 167 and public health practitioners. One of the most important lessons we learned is that our 168 successes required a large team with experience across disciplines. The urgency of the 169 pandemic allowed us to motivate a singular, and often uncompensated, effort to quickly curate 170 and evaluate emerging science. This team-based approach allowed the NCRC to not only code 171 a complex content management system from scratch, but also to cover a wide variety of topics 172 from transmission across settings, mask-wearing behaviors, and the use of dogs for 173 detecting SARS-CoV-2 to vaccine efficacy against SARS-CoV-2 variants. 174 for use under a CC0 license. This article is a US Government work. It is not subject to copyright under 17 USC 105 and is also made available (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. SARS-CoV-2. However, there are important risks of using data from preprints to make clinical 178 and public health decisions. As these papers have not yet undergone peer review, they are 179 often more difficult to parse and may contain flaws in their methodology or analyses that could 180 fatally compromise the findings and conclusions. Given the increasing attention from the media 181 on preprints and the urgency to quickly understand the novel coronavirus, the NCRC designed 182 its system to routinely include preprints in the review process. The NCRC's collaboration with 183 bioRxiv and medRxiv ensures that reviews of preprints are seen by authors and others, like 184 journalists or the public, who might access them directly from the preprint server. 185 The NCRC's team of experts aimed to quickly identify popular but flawed articles. Even after 186 peer-review, errors in analysis or overreach in the interpretation of data in research studies may 187 be identified post-publication. The extraordinary speed of publication during the SARS-CoV-2 188 pandemic may have exacerbated the risk of these errors. Sometimes, errors can give rise to 189 surprising results, and these can garner outsized attention due to the claims they make. On 190 multiple occasions, the NCRC posted reviews to critically evaluate whether the conclusions of a 191 paper were supported by its data and to highlight key methodological shortcomings when 192 applicable to separate actionable evidence from questionable evidence. The NCRC effort to 193 offer critiques in a single place provides practitioners a convenient resource to cut through the 194 noise. 195 The structure of the NCRC also provides a crucial training opportunity for doctoral students and 196 postdoctoral fellows to understand the connections between academic literature and practice. 197 communication, faculty also meet regularly with students and fellows to reflect on which papers 200 should be included on the NCRC website. The curation process helps to hone the skills of 201 students and fellows to understand the current landscape, gaps in knowledge, and which 202 research questions are indispensable for moving the evidence base forward. 203 Despite these strengths, the sustainability of such a large effort is unclear. In particular, the 204 NCRC has been unable to identify long-term funding to cover the costs of faculty time dedicated 205 to the project and this poses a risk to the future of the endeavor. The time lag between paper 206 publication and our reviews has been a concern, since it has reduced our ability to contribute to 207 discussions about evidence in real time. These delays had multiple contributors, including the 208 lags between publication and indexing in PubMed, but a lack of salary support for the effort also 209 contributed. Our ability to measure the impact of the NCRC on changes to knowledge or 210 practice among our target audience has been limited, for multiple reasons. Any impact 211 measurement would almost certainly require interviews with members of our target audience, 212 who currently have no time to spare from pandemic response to participate in such endeavors. 213 Additionally, the NCRC team has been stretched to keep up with the literature leaving little time 214 to focus on impact assessments, though future work focused on outcome assessment should 215 be considered. Nevertheless, continued utilization of the reviews, reflected in growing newsletter 216 subscribers and increases in webpage views, and informal conversations with colleagues in our 217 target audience suggest that the NCRC is a valued resource. 218 As the COVID-19 pandemic has emphasized the necessity of collaboration between public 219 health researchers and practitioners to save lives, we believe that the NCRC is one model of 220 how this collaboration can be successful. Since its inception, we have conceptualized the NCRC 221 as a critical training opportunity for the next generation of public health and clinical scientists in 222 knowledge translation to improve decision making. The NCRC approach could be useful to 223 improve timely translation of data to action for other public health problems where lags in 224 for use under a CC0 license.
This article is a US Government work. It is not subject to copyright under 17 USC 105 and is also made available (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. This article is a US Government work. It is not subject to copyright under 17 USC 105 and is also made available (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. This article is a US Government work. It is not subject to copyright under 17 USC 105 and is also made available (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.
The copyright holder for this preprint this version posted April 27, 2021. ; This article is a US Government work. It is not subject to copyright under 17 USC 105 and is also made available (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.  This article is a US Government work. It is not subject to copyright under 17 USC 105 and is also made available (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.