Potential Test-Negative Case-Control Study Bias in Outbreak Settings: Application to Ebola vaccination in Democratic Republic of Congo ====================================================================================================================================== * Carl A. B. Pearson * W. John Edmunds * Thomas J. Hladish * Rosalind M. Eggo ## Abstract **Background** Infectious disease outbreaks present unique challenges to study designs for vaccine evaluation. Test-negative case-control (TNCC) studies have been used to estimate vaccine efficacy previously, and have been proposed for Ebola virus disease (EVD) vaccines. However, there are key differences in how cases and controls are recruited during outbreaks that have implications for the reliability of vaccine efficacy estimates from these studies. **Methods** We use a modelling approach to quantify TNCC bias for a prophylactic vaccine distributed across varying study and epidemiological scenarios. Our model accounts for vaccine distribution heterogeneity and for the two potential routes of recruitment: self-reporting and contact-tracing. We derive the TNCC estimator for this model and suggest ways to translate outbreak response data into the parameters of the model. **Results** We found systematic biases in vaccine estimates from a TNCC study in our model of outbreak conditions. Biases are introduced due to differential recruitment from self-report and contact-tracing, and by clustering of participation in vaccination. We estimate the magnitude of these biases, and highlight options to manage them via restricted recruitment. For the motivating example of EVD, the absolute bias should be less 10%. **Conclusions** A TNCC study may generate biased estimates of vaccine efficacy during outbreaks. Bias can be limited via recruitment that either minimizes heterogeneity in vaccination in the recruited population or excludes recruitment of contact-traced individuals. TNCC studies for outbreak infections should record the reason for testing to quantify potential bias in the vaccine efficacy estimate. Perfectly distinguishing the recruitment route may be difficult in practice, so it will be challenging to entirely remove this bias. **Key Messages** * - Test-Negative Case-Control (TNCC) studies can be biased when follow-up of cases leads to recruitment, which may happen during outbreak response. * - The absolute bias can be quantified using epidemiological measures. * - Bias can be limited by ensuring homogeneous vaccine coverage amongst potential recruits. * - Bias can be eliminated if excluding recruits from contact-tracing is practical. * - Based on assumptions about the outbreak in Nord Kivu DRC starting in 2018, a TNCC vaccine study there would have a maximum absolute bias of less than 10% due to these effects. ## Background Study designs to evaluate vaccine efficacy for new vaccines during outbreaks are challenging for logistical, epidemiological, social, and ethical reasons (1-7). Outbreaks can overwhelm local health systems, subsequently enabling other crises, complicating both response efforts and research (8-11). However, some key pathogens are only routinely observable under these conditions, like Ebola virus disease (EVD) and others on the World Health Organization (WHO) R&D Blueprint priority list (12). During outbreaks of highly pathogenic infections there may be pressure to introduce experimental vaccines as quickly as possible (13, 14), as well as resistance to classical randomised controlled trials (15). For EVD, the existence of a licensed vaccine (16-19) further complicates trials for new vaccines. Such circumstances require alternative evaluation strategies, and the test-negative case-control (TNCC) design has been proposed to evaluate a two-dose vaccine in eastern Democratic Republic of Congo (DRC) (20-22). This design estimates vaccine efficacy from the odds ratio for test outcome conditional on vaccination status, and has lower misclassification bias than traditional case-control studies (23, 24). TNCC studies have been used estimate the efficacy of vaccines against influenza (25), rotavirus (26-28), pneumococcus (29), and other pathogens (30, 31). The design can also assess interventions such as vector control and risk-factor management (32-34). TNCC studies recruit people with symptoms, test those recruits using a highly sensitive and specific method to separate cases (test-positives) from controls (test-negatives), and finally sort them by vaccination status (35). These TNCC studies can be retrospective, potentially using stratification by other factors to limit confounding. For influenza, TNCC studies usually recruit people seeking care for influenza-like illness, ascertain vaccine status by self-report, and determine infection status by RT-PCR, though specifics vary (25, 30). To obtain unbiased estimates, the following criteria must be met: i) transmission occurs in a population with partial vaccine coverage; ii) test-negative rates are unaffected by vaccination status; iii) given symptoms in an individual, care-seeking behaviour does not vary by underlying cause; iv) an individual’s past recruitment as a control (even multiple times) must not prevent subsequent recruitment as a case; and v) there is no misclassification of individuals’ infection or vaccine status (33, 36). We examined how outbreaks present novel misclassification problems for TNCC studies, and how this can bias TNCC vaccine efficacy estimates. We quantified how that bias varies with differences in vaccine distribution, recruitment, risk of infection, and testing practice, and we identified steps to mitigate this bias. ## Methods ### Key Differences During EVD Outbreaks EVD tests are used both to make treatment decisions for individuals and to trigger public health responses to the outbreak (*e*.*g*. post-mortem testing). During EVD outbreaks, suspect cases are tested for distinct reasons: either presence of multiple symptoms or contact with a known case, plus fever (37, 38). We treat these testing reasons as distinct recruiting sources: people who self-report seeking care for EVD-like symptoms (analogous to influenza studies); and people identified via active contact-tracing from a confirmed case. Depending on how data are collected, individual testing reasons may be unavailable. For clarity, we provide specific definitions for our model terms (**Table 1**). View this table: [Table 1:](http://medrxiv.org/content/early/2020/01/10/2020.01.06.20016576/T1) Table 1: Definitions of terms used in this analysis. ### Vaccination Model Vaccinated individuals avoid infection if exposed to EVD with probability *E*, the true vaccine efficacy. The study measures *Ê*, the estimated vaccine efficacy. Vaccines may have further complications, such as multiple doses or a delay to protection, but we model a scenario where protection has already occurred in the recruitable population. Aside from preventing EVD, vaccination has no effect. We represent vaccine distribution by dividing the recruitable population into two types of individuals: those who participate in the vaccination campaign and those who do not (Figure 1). They are participating and non-participating (Table 1), and the fraction is *p**in* (Table 2). Among participating individuals, only some receive the vaccine, leading to vaccine coverage level, *L*. We assume individuals cluster by participation status, so the contacts of self-reporting cases always have the same participation status as the associated case. Aside from these distinctions, all individuals are identical. View this table: [Table 2:](http://medrxiv.org/content/early/2020/01/10/2020.01.06.20016576/T2) Table 2: Parameter Summary. This table summarizes the measurements and model parameters used in this analysis. We also introduce an alternative parameterization of the recruitment model, which is less intuitive when describing the model but more useful for understanding the impact on bias. ![Figure 1.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2020/01/10/2020.01.06.20016576/F1.medium.gif) [Figure 1.](http://medrxiv.org/content/early/2020/01/10/2020.01.06.20016576/F1) Figure 1. The modelled population and recruitment into the TNCC study. (a) Individuals and their contacts either participate in vaccination (filled circles) or do not (open circles). (b) The fraction that participate (and thus may be vaccinated) is pin. Of those that participate, some are not vaccinated (e.g. because they are ineligible) (light blue) and some are vaccinated (dark blue). The vaccine coverage in participants is L. In the recruitable population, non-vaccinees are infected on EVD exposure, while vaccinees avoid disease at the vaccine efficacy, E. (c) An expected number of self-reported people test negative, B, until a test-positive is identified. This leads to an expected amount of follow-up testing, λ, which finds R” more cases if the initial case is non-participating, and (1-LE)R” if participating. The coverage, L, efficacy, E, and participating fraction, pin, determine the likelihood of observing the self-reporting case among participating vs non-participating individuals and vaccinated vs unvaccinated individuals. (d) Resulting categories that can be recruited into the study. Ideally, a study would recruit only participating individuals, but they may not be distinguishable in practice. Indeed, in reality there may be many distinct populations, for example areas with different vaccination coverage; we consider just two to focus on the impact of heterogeneity. ### TNCC Recruitment Model We identify recruits by their testing route, either self-reported or contact-traced from a confirmed case (**Table *1*** and Figure 1). In the Supplement, we generalise these as *primary* and *secondary* recruitment routes. We assume that introduced cases in the recruitable population are found before any contacts have become symptomatic and therefore infectious (though they may be infected but not yet symptomatic), consistent with the typical experience in the rVSV trial in DRC (39). Thus, in the model all self-reported cases result from external introductions, and all contact-traced recruits are only exposed to a one case. Consistent with the field alerts process, we model self-reporting as presenting with multiple EVD-like symptoms caused either by EVD (a self-reported test-positive) or not (a self-reported test-negative). Participating and non-participating individuals are assumed to have equal rates of EVD and non-EVD exposure, but on average participating individuals have fewer EVD infections due to the vaccine. During an outbreak, EVD and non-EVD exposure rates may vary, but our analysis only depends on the long-term average relative rate of these processes. We can then define parameters as per self-reported test-positive, and we use the background self-reporting test-negatives per test-positive rate, *B*, to represent the self-reporting process overall (Table 2). We also represent contact-traced testing based on the field alerts process. Fever is common in EVD-prone areas and we therefore assume that meeting this criterion and testing contacts is frequent. The average number of tested contacts is λ, which is the same irrespective of EVD exposure or participation status (Supplement Section S2.3 relaxes this assumption). ### Translating outbreak metrics to estimate bias To evaluate a particular study’s potential bias, we need real-world outbreak response metrics to estimate model parameters. For studies augmenting an ongoing outbreak response, data already collected could be used. For example, data on the number of tested individuals, stratified by test outcome and testing route could be used to bound model parameters (Table 2). The model also depends on how the study vaccination is distributed, represented by participating fraction and coverage, *p**in* and *L*. Depending on the study protocol, these could be ascertained in different ways (Supplement Section S7). ### Summary of assumptions * Cases and their contacts have the same participation status. * All individuals have the same exposure risk to EVD and other causes of EVD-like symptoms, average number of contacts, and risk of infection per contact. * Non-vaccination among participating individuals happens randomly. * There are different testing criteria for self-reporting and contact-tracing individuals. * Self-reporting cases are identified before anyone they have infected becomes symptomatic, and contact tracing prevents transmission amongst contacts. ### TNCC Estimator for Outbreak Context There are twelve recruitment categories in our model, based on participation and vaccination status, test outcome, and testing route (Figure 1). The conventional TNCC estimator: ![Formula][1] can be written with these categories as: ![Formula][2] with participation status by letter, testing route by superscript, and test outcome by subscript (Figure 1). The expected counts of these categories can be expressed with the six model parameters we defined (full derivation in Supplement S3-S4): ![Formula][3] The terms to the right only cancel under very specific circumstances, thus the bias is generally non-zero and the magnitude includes all model parameters. We refactor Eq. 2 with alternative parameters relating to recruitment and epidemiological measures relating to the outbreak (Supplement Section S4). We use this form to explore the bias and to evaluate potential maximum bias under specific outbreak scenarios: ![Formula][4] ### TNCC Estimator Bias Across a wide range of self-reporting test-negative fractions (*f*−), contact-tracing test-positive fractions (*p**t*), participating fractions (*p**in*), and recruitment route ratios (*ρ*), the absolute error in *Ê* is ≤ 0.1 (Figure 2). If information from the outbreak response indicates these parameter values are reasonable, then our model indicates study bias lies in that range. If that level of bias is not acceptable, or if the study or outbreak parameters are outside of this range, we have identified two avenues to limit bias, either restricting recruitment to participating or self-reporting individuals only. ![Figure 2:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2020/01/10/2020.01.06.20016576/F2.medium.gif) [Figure 2:](http://medrxiv.org/content/early/2020/01/10/2020.01.06.20016576/F2) Figure 2: Bias Trends Across All Model Parameters. The figure illustrates the bias trends with respect to true efficacy, *E*, and vaccination coverage among participating individuals, *L*. The sixteen panels correspond to combinations of example values for: (outer columns) self-reporting test-negative fraction (*f*− at low = 0.8 and high = 0.99); (inner columns) the recruitment route ratio (ρ at low = 0.5 and high = 2; less than 1 implies more self-reporting recruitment, greater than 1 implies more contact-tracing recruitment); (outer rows) contact-tracing test-positive fraction (*p**t* at low = 0.1 and high = 0.3); and the participating fraction (pin, at low = 0.6 and high = 0.9). ### Restricting Recruitment to Participating Individuals Ideally, a study would achieve high participation in the recruitable population. This may be possible if community engagement increases participation, or if there is additional data collected that allows exclusion of non-participants, though identification of these individuals may be difficult. Even if the study is constrained to only participating individuals, *i*.*e*. so *p**in* = 1, some bias remains due to recruitment of contact-traced individuals but it no longer depends on the vaccine coverage (see Supplement Section S5.1). However, perfect participation is unlikely, and the bias is sensitive to other factors even when participation is high (Figure 3). ![Figure 3:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2020/01/10/2020.01.06.20016576/F3.medium.gif) [Figure 3:](http://medrxiv.org/content/early/2020/01/10/2020.01.06.20016576/F3) Figure 3: Impact of Decreasing Participating Fraction Amongst Recruits. The panels show decreasing participation fraction (columns from left to right) for scenarios stratified by self-reported test-negative fraction in recruitment (0.8 and 0.99) and recruitment route ratio (0.1 and 0.3) (rows). This figure shows 70% coverage level among participating individuals, *L*=0.7. If the participating fraction decreases, generally the magnitude of bias in the estimate increases. Bias generally peaks when the true efficacy is around 50%, and goes to zero as efficacy goes to 0 or 100%. Because initial cases are more likely to be non-participating individuals, contact-traced individuals from those cases are biased towards non-participation as well. Thus, bias trends toward overestimation as contact-traced individuals more frequently test positive. This can reverse for high levels of contact-traced recruitment, when most contact-traced individuals are test-negative. All other factors being equal, more coverage means more extreme bias as participating and non-participating individuals diverge. ### Restricting Recruitment to Self-Reported individuals The bias can also be corrected by restricting recruitment to self-reported individuals only. If perfectly achieved then the bias is 0 (see Supplement S5.2). However, while there is a clear need to track cases during an outbreak response, there is less motivation to carefully monitor test-negatives. Thus, it may be possible to distinguish testing routes for test-positives, but not for test-negatives. If that applies, the resulting estimator bias is: ![Formula][5] Minimising, and ultimately eliminating bias in Equation (4) still depends on maximising *p**in*. One way of achieving high *p**in* is excluding non-participants; this suggests a trade-off between precision and bias; if the study invests the effort to exclude non-participants it may make sense to include contact-traced recruits. If contact-traced test-positives are excluded retrospectively, the resulting bias magnitude can be lower even if contact-traced test-negatives are included due to misclassification (Figure 3 vs Figure 4). However, the direction of bias changes with changing participation rates: the neutral line falls at higher contact-tracing test-positive fractions when participation decreases. The magnitude of bias at the extremes of the contact-tracing test-positive fractions is driven largely by the amount of self-reporting test-negatives. Other factors being equal, fewer self-reported test-negatives means a lower self-reporting test-negative fraction (0.8 instead of 0.99) and higher recruitment route ratio (0.3 instead of 0.1), both of which correspond to more extreme bias. ![Figure 4:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2020/01/10/2020.01.06.20016576/F4.medium.gif) [Figure 4:](http://medrxiv.org/content/early/2020/01/10/2020.01.06.20016576/F4) Figure 4: Bias Due to Inability to Exclude Contact-Traced Test-Negatives. The panels show decreasing participation fraction (columns from left to right) for scenarios stratified by self-reported test-negative fraction in recruitment (0.8 and 0.99) and recruitment route ratio (0.1 and 0.3) (rows). This figure shows 70% coverage level among participating individuals, *L*=0.7. The range of bias is usually smaller than when recruitment is restricted to the participating individuals only (Figure 3). Quantifying Potential Bias from Outbreak Response Metrics To quantify the range of bias for a study in DRC, we determined the plausible range of outbreak response metrics and corresponding model parameters, where SR+ was 100-150, CT+ was 100-400, SR− was 6500-7000, and CT− was 900-1200. When restricting recruitment to participating individuals only, the bias in vaccine efficacy is less than 3% overestimation (Figure 5, left panel), but can increase up to >15% overestimate for high coverage and low participation (Figure 5, right panel; 90% coverage, 40% participation). As more non-participating individuals are included, increasing coverage increases bias, corresponding to the increasing distinction in infection risk between participation groups. For these outbreak response metrics, the estimate of vaccine efficacy consistently exceeds the true efficacy. ![Figure 5:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2020/01/10/2020.01.06.20016576/F5.medium.gif) [Figure 5:](http://medrxiv.org/content/early/2020/01/10/2020.01.06.20016576/F5) Figure 5: Bias possible when recruiting participating individuals only. These bias envelopes were computed assuming outbreak response metrics SR− ∈ (6500, 7000), SR+ ∈ (100, 150), CT+ ∈ (100, 400), and CT− ∈ (900, 1200), which corresponds to 97.7-98.6% of self-reporting cases testing negative, testing 6-16 contact-traced individuals per self-reported case, and 10-25% of those contact-traced individuals testing positive. If the study is restricted to recruit only the participating individuals (left-most panel), then bias can be limited to less than 3% overestimation. However, as the participating fraction falls, the error range generally increases, to >15% peak bias for high coverage (90%) and low participation (40%). Higher coverage among participating individuals generally increases bias; this reflects increasing differences between the participating and non-participating individuals. When restricting recruitment to self-reported individuals only, with no misclassification of testing route for test-negatives, the bias is 0 (Figure 6, left panel). As the misclassification rate increases from 0 to 100%, the bias increases and tends toward underestimation, though the range of possible bias includes overestimation. For the most extreme case, including all of contact-traced test-negatives for a low participation fraction and a high-coverage, the bias spans roughly 1% overestimate to 5% underestimate (Figure 6, right panel; 90% coverage, 40% participation). As with restricting recruitment to participating individuals, the magnitude of the bias increases with coverage amongst participating individuals. ![Figure 6:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2020/01/10/2020.01.06.20016576/F6.medium.gif) [Figure 6:](http://medrxiv.org/content/early/2020/01/10/2020.01.06.20016576/F6) Figure 6: Bias possible when recruitment is restricted to self-reported individuals only. If the study analysis is able to restrict recruits to only self-reporting individuals, then there is no bias (left panel). However, as contact-traced test-negative individuals are increasingly included (moving right across panels), bias range increases to between 1% overestimate and 5% underestimate. However, this range is notably smaller than if only restricting to participating recruits (Figure 5). As with restricting recruitment to participating individuals only, higher levels of coverage lead to wider bias range. These ranges reflect the same parameters used in Figure 5, including participating fraction *p**in* ∈ (0.4, 1). ## Discussion Previous work has explored biases in TNCC studies due to care-seeking or other confounding and selection effects (35, 40, 41), test or vaccine status misclassification errors (25, 42-44), and vaccine mechanism (36, 45). We demonstrated that the particular epidemiology of an outbreak can also generate bias in the vaccine efficacy estimates in a TNCC study. If a study cannot distinguish self-reported and contact-traced recruits or whether recruits were generally amongst a vaccinating group, then the vaccine efficacy estimate is potentially biased. These are both real, practical problems: in addition to general difficulty of collecting data during an outbreak, it may be difficult to achieve uniform levels of vaccine coverage when rolling out a vaccine in an emergency setting, particularly in highly mobile populations or those affected by civil unrest. The bias in our model arises from the interaction of heterogeneous vaccination distribution and the inclusion of tested individuals from contact-tracing. Because initial cases found through self-reporting are more likely to be non-participating individuals (and thus non-vaccinating), including contact-traced recruits overrepresents those individuals in the estimator. This in turn can tilt the bias either towards or away from null, depending on how infection risk and testing criteria in the contact-traced individuals differs from that in the general population. If the self-reporting process leads to many more test-negative recruits than recruits from contact-tracing (either positive or negative), then the bias from contact-tracing is relatively smaller. If it does not, then the relative number of cases versus controls from contact-tracing will determine the general direction of bias; more cases will lead to overestimation, more controls to underestimation. We showed that the range of potential bias can be quantified with some epidemiological data from the outbreak. For the range of outbreak response metrics we used to represent the EVD outbreak in DRC, this is less than 10% if the study can achieve high participation (*p**in* ≥ 0.6) with moderate coverage (*L* ≤ 0.7). Practically, it may be possible to limit but not eliminate these drivers of the bias. For the EVD epidemic in eastern DRC, responders try to test individuals that meet one of the outbreak “suspected case” definitions, which combine different levels of symptoms and potential contact with known cases (37, 38). This practice would likely continue in populations that received a study vaccine, meaning this testing process would be the likely source for a retrospective TNCC study of a new vaccine. We framed our analysis in terms of event counts, but it can also be thought of in terms of testing thresholds. For example, we frame contact-tracing recruitment as a number of contacts and the number of cases amongst them, and the resulting efficacy estimator error is driven by the ratio of those values. In an infectious disease sense, this ratio is the transmission probability, but it could instead be interpreted as precision of the criteria for testing (*i*.*e*. positive-predictive value): are the alert criteria such that we test fewer contacts and a larger percentage are positive (higher precision) or do we test more frequently to ensure no positives are missed (lower precision)? A similar analogy applies for testing outside contact-tracing. During an outbreak, the response may prefer less stringent testing criteria to ensure as many cases as possible are identified to aid control. For both the self-reported and contact-traced testing, the vaccine efficacy estimate bias is generally lower when testing criteria are less stringent in this sense, consistent with a control-oriented outbreak response. Our analysis describes vaccination during EVD outbreaks, but our work has general implications for evaluating interventions in other outbreak settings. We have focused on self-reporting and contact-tracing, but the challenges are generic when there are distinct but potentially indistinguishable primary and secondary recruiting processes. For example, more active general-population surveillance could still qualify as the primary recruitment in our model, as long as it was random with respect to intervention status. Likewise, geographic follow-up could be a secondary process, as long as intervention status was correlated with the secondary process (*e*.*g*. for dengue, adjacent households followed up, as long as vector control reached some areas and not others). This analysis of the TNCC design under outbreak conditions does not consider other possible sources of bias, such as different exposure risk between groups, testing errors, errors in ascertaining vaccine status especially when there may be multiple vaccines or a long period between vaccination and exposure, or other data problems that could occur. Further investigation of the reliability of TNCC studies for vaccine efficacy estimation during outbreaks remains critical. However, as we have shown, use of this design in an outbreak setting will need to account for the realities of control activities, and plan to collect data on testing route or otherwise accommodate the mix of recruitment routes. Adoption of a TNCC design to evaluate a new vaccine in DRC may increase pressure for similar studies that do not have an explicit, randomised control group in future outbreaks of similarly highly pathogenic diseases. Understanding the biases and limitations of the TNCC design will therefore be critical to evaluate vaccines that are currently being developed against these pathogens. ## Data Availability The results are simulation-based. All code for those simulations can be found in the repository. [https://gitlab.com/ebovac/tncc\_math](https://gitlab.com/ebovac/tncc_math) ## Acknowledgements We thank Peter G. Smith and Nick Jewell for useful discussions during the development of this model and analysis. This work was supported by the Innovative Medicines Initiative 2 (IMI2) Joint Undertaking between European Union Horizon 2020 Research and Innovation Programme and the European Federation of Pharmaceutical Industries and Associations [EBOVAC3: grant number 800176]; Department of Health and Social Care using UK Aid funding managed by the NIHR [VEEPED: PR-OD-1017-20007]; National Institutes of Health, National Institute of General Medical Sciences [U54 GM111274]; and HDR UK Innovation Fellowship [grant MR/S003975/1 to R.M.E.]. The views expressed in this publication are those of the authors and not necessarily those of the Department of Health and Social Care. ## Footnotes * * carl.pearson{at}lshtm.ac.uk * Received January 6, 2020. * Revision received January 6, 2020. * Accepted January 10, 2020. * © 2020, Posted by Cold Spring Harbor Laboratory The copyright holder for this pre-print is the author. All rights reserved. The material may not be redistributed, re-used or adapted without the author's permission. ## References 1. 1.Halloran ME, Auranen K, Baird S, Basta NE, Bellan SE, Brookmeyer R, et al. Simulations for designing and interpreting intervention trials in infectious diseases. BMC Med. 2017;15(1):223. 2. 2.Bellan SE, Eggo RM, Gsell P-S, Kucharski AJ, Dean NE, Donohue R, et al. An online decision tree for vaccine efficacy trial design during infectious disease epidemics: The InterVax-Tool. Vaccine. 2019;37(31):4376–81. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.vaccine.2019.06.019&link_type=DOI) 3. 3.Dean NE, Gsell P-S, Brookmeyer R, De Gruttola V, Donnelly CA, Halloran ME, et al. Considerations for the design of vaccine efficacy trials during public health emergencies. bioRxiv. 2018. 4. 4.Baden LR, Rubin EJ, Morrissey S, Farrar JJ, Drazen JM. We Can Do Better - Improving Outcomes in the Midst of an Emergency. N Engl J Med. 2017;377(15):1482–4. 5. 5.WHO. Vaccination in acute humanitarian emergencies: a framework for decision making. 2017. 6. 6.Moodley K, Hardie K, Selgelid MJ, Waldman RJ, Strebel P, Rees H, et al. Ethical considerations for vaccination programmes in acute humanitarian emergencies. Bull World Health Organ. 2013;91(4):290–7. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.2471/BLT.12.113480&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=23599553&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F01%2F10%2F2020.01.06.20016576.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000317889800011&link_type=ISI) 7. 7.Hatchett R, Lurie N. Outbreak response as an essential component of vaccine development. The Lancet Infectious Diseases. 2019;19(11):e399–e403. 8. 8.Owada K, Eckmanns T, Kamara KB, Olu OO. Epidemiological Data Management during an Outbreak of Ebola Virus Disease: Key Issues and Observations from Sierra Leone. Front Public Health. 2016;4:163. 9. 9.Kieny MP, Salama P. WHO R&D Blueprint: a global coordination mechanism for R&D preparedness. The Lancet. 2017;389(10088):2469–70. 10. 10.Hofman M, Au S. The politics of fear: Medecins sans frontieres and the West African Ebola epidemic: Oxford University Press; 2017. 11. 11.Beavogui AH, Delamou A, Yansane ML, Konde MK, Diallo AA, Aboulhab J, et al. Clinical research during the Ebola virus disease outbreak in Guinea: Lessons learned and ways forward. Clin Trials. 2016;13(1):73–8. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1177/1740774515619877&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=26768557&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F01%2F10%2F2020.01.06.20016576.atom) 12. 12.Mehand MS, Al-Shorbaji F, Millett P, Murgue B. The WHO R&D Blueprint: 2018 review of emerging infectious diseases requiring urgent research and development efforts. Antiviral Res. 2018;159:63–7. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.antiviral.2018.09.009&link_type=DOI) 13. 13.Kupferschmidt K. Plan to use second Ebola vaccine sparks debate. Science. 2019;364(6447):1221-. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6Mzoic2NpIjtzOjU6InJlc2lkIjtzOjEzOiIzNjQvNjQ0Ny8xMjIxIjtzOjQ6ImF0b20iO3M6NTA6Ii9tZWRyeGl2L2Vhcmx5LzIwMjAvMDEvMTAvMjAyMC4wMS4wNi4yMDAxNjU3Ni5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 14. 14.Independent Ebola vaccination committee is needed to overcome lack of WHO transparency [press release]. Medecins Sans Frontieres, 23-09-2019. 15. 15.Adebamowo C, Bah-Sow O, Binka F, Bruzzone R, Caplan A, Delfraissy J-F, et al. Randomised controlled trials for Ebola: practical and ethical issues. The Lancet. 2014;384(9952):1423–4. 16. 16.Consortium EçSRVT. The ring vaccination trial: a novel cluster randomised controlled trial design to evaluate vaccine efficacy and effectiveness during outbreaks, with special reference to Ebola. BMJ (Clinical research ed). 2015;351:h3740. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjE3OiIzNTEvanVsMjdfNy9oMzc0MCI7czo0OiJhdG9tIjtzOjUwOiIvbWVkcnhpdi9lYXJseS8yMDIwLzAxLzEwLzIwMjAuMDEuMDYuMjAwMTY1NzYuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 17. 17.Henao-Restrepo AM, Camacho A, Longini IM, Watson CH, Edmunds WJ, Egger M, et al. Efficacy and effectiveness of an rVSV-vectored vaccine in preventing Ebola virus disease: final results from the Guinea ring vaccination, open-label, cluster-randomised trial (Ebola Ça Suffit!). The Lancet. 2017;389(10068):505–18. 18. 18.Vaccine against Ebola: Commission grants first-ever market authorisation [press release]. European Commission, 11 November 2019. 19. 19.First FDA-approved vaccine for the prevention of Ebola virus disease, marking a critical milestone in public health preparedness and response [press release]. FDA, 19 December 2019. 20. 20.WHO. Weekly epidemiological record. World Health Organization; 2019. p. 261–80. 21. 21.SAGE. Strategic Advisory Group of Experts (SAGE) on Immunization Interim Recommendations on Vaccination against Ebola Virus Disease (EVD). World Health Organization (WHO); 2019. 22. 22.Johnson & Johnson Announces Donation of up to 500,000 Regimens of Janssen’s Investigational Ebola Vaccine to Support Outbreak Response in Democratic Republic of the Congo (DRC) [press release]. 2019. 23. 23.Vandebosch A, Mogg R, Goeyvaerts N, Truyers C, Greenwood B, Watson-Jones D, et al. Simulation-guided phase 3 trial design to evaluate vaccine effectiveness to prevent Ebola virus disease infection: Statistical considerations, design rationale, and challenges. Clin Trials. 2016;13(1):57–65. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1177/1740774515621059&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=26768568&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F01%2F10%2F2020.01.06.20016576.atom) 24. 24.ECBS ECOBS. Guidelines on the quality, safety and efficacy of Ebola vaccines. WHO Technical Report Series. 2017;68:87–179. 25. 25.Sullivan SG, Feng S, Cowling BJ. Potential of the test-negative design for measuring influenza vaccine effectiveness: a systematic review. Expert Rev Vaccines. 2014;13(12):1571–91. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1586/14760584.2014.966695&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=25348015&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F01%2F10%2F2020.01.06.20016576.atom) 26. 26.Araki K, Hara M, Shimanoe C, Nishida Y, Matsuo M, Tanaka K. Case-Control Study of Rotavirus Vaccine Effectiveness Compared to Test-Negative Controls or Hospital Controls. J Epidemiol. 2019;29(8):282–7. 27. 27.Walker JL, Andrews NJ, Atchison CJ, Collins S, Allen DJ, Ramsay ME, et al. Effectiveness of oral rotavirus vaccination in England against rotavirus-confirmed and all-cause acute gastroenteritis. Vaccine X. 2019;1:100005. 28. 28.Haber M, Lopman BA, Tate JE, Shi M, Parashar UD. A comparison of the test-negative and traditional case-control study designs with respect to the bias of estimates of rotavirus vaccine effectiveness. Vaccine. 2018;36(33):5071–6. 29. 29.Broome CV, Facklam RR, Fraser DW. Pneumococcal disease after pneumococcal vaccination: an alternative method to estimate the efficacy of pneumococcal vaccine. N Engl J Med. 1980;303(10):549–52. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1056/NEJM198009043031003&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=6995835&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F01%2F10%2F2020.01.06.20016576.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=A1980KE80100003&link_type=ISI) 30. 30.Chua H, Feng S, Lewnard JA, Sullivan SG, Blyth CC, Lipsitch M, et al. The use of test-negative controls to monitor vaccine effectiveness: a systematic review of methodology. Epidemiology. 2019. 31. 31.Li Y, Zhou Y, Cheng Y, Wu P, Zhou C, Cui P, et al. Effectiveness of EV-A71 vaccination in prevention of paediatric hand, foot, and mouth disease associated with EV-A71 virus infection requiring hospitalisation in Henan, China, 2017–18: a test-negative case-control study. The Lancet Child & Adolescent Health. 2019;3(10):697–704. 32. 32.Yung CF, Chan SP, Thein TL, Chai SC, Leo YS. Epidemiological risk factors for adult dengue in Singapore: an 8-year nested test negative case control study. BMC Infect Dis. 2016;16:323. 33. 33.Anders KL, Cutcher Z, Kleinschmidt I, Donnelly CA, Ferguson NM, Indriani C, et al. Cluster- Randomized Test-Negative Design Trials: A Novel and Efficient Method to Assess the Efficacy of Community-Level Dengue Interventions. Am J Epidemiol. 2018;187(9):2021–8. 34. 34.Anders KL, Indriani C, Ahmad RA, Tantowijoyo W, Arguni E, Andari B, et al. The AWED trial (Applying Wolbachia to Eliminate Dengue) to assess the efficacy of Wolbachia-infected mosquito deployments to reduce dengue incidence in Yogyakarta, Indonesia: study protocol for a cluster randomised controlled trial. Trials. 2018;19(1):302. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1186/s13063-018-2670-z&link_type=DOI) 35. 35.Foppa IM, Haber M, Ferdinands JM, Shay DK. The case test-negative design for studies of the effectiveness of influenza vaccine. Vaccine. 2013;31(30):3104–9. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.vaccine.2013.04.026&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=23624093&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F01%2F10%2F2020.01.06.20016576.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000321178800011&link_type=ISI) 36. 36.Dean NE. RE: “MEASUREMENT OF VACCINE DIRECT EFFECTS UNDER THE TEST-NEGATIVE DESIGN”. American Journal of Epidemiology. 2019;188(4):806–10. 37. 37.WHO. Case definition recommendations for Ebola or Marburg virus diseases. WHO; 2014. 38. 38.WHO. Implementation and management of contact tracing for Ebola virus disease. WHO; 2015. 39. 39.WHO. Preliminary results on the efficacy of rVSV-ZEBOV-GP Ebola vaccine using the ring vaccination strategy in the control of an Ebola outbreak in the Democratic Republic of the Congo: an example of integration of research into epidemic response. 2019. 40. 40.Ainslie KEC, Shi M, Haber M, Orenstein WA. On the bias of estimates of influenza vaccine effectiveness from test-negative studies. Vaccine. 2017;35(52):7297–301. 41. 41.Ferdinands JM, Foppa IM, Fry AM, Flannery BL, Belongia EA, Jackson ML. Re: “Invited Commentary: Beware the Test-Negative Design”. Am J Epidemiol. 2017;185(7):613. 42. 42.Orenstein EW, De Serres G, Haber MJ, Shay DK, Bridges CB, Gargiullo P, et al. Methodologic issues regarding the use of three observational study designs to assess influenza vaccine effectiveness. Int J Epidemiol. 2007;36(3):623–31. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1093/ije/dym021&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=17403908&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F01%2F10%2F2020.01.06.20016576.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000249098500031&link_type=ISI) 43. 43.Jackson ML, Rothman KJ. Effects of imperfect test sensitivity and specificity on observational studies of influenza vaccine effectiveness. Vaccine. 2015;33(11):1313–6. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.vaccine.2015.01.069&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=25659280&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F01%2F10%2F2020.01.06.20016576.atom) 44. 44.De Smedt T, Merrall E, Macina D, Perez-Vilar S, Andrews N, Bollaerts K. Bias due to differential and non-differential disease- and exposure misclassification in studies of vaccine effectiveness. PLoS One. 2018;13(6):e0199180. 45. 45.Lewnard JA, Tedijanto C, Cowling BJ, Lipsitch M. Measurement of Vaccine Direct Effects Under the Test-Negative Design. American Journal of Epidemiology. 2018;187(12):2686–97. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1093/aje/kwy163&link_type=DOI) [1]: /embed/graphic-4.gif [2]: /embed/graphic-5.gif [3]: /embed/graphic-6.gif [4]: /embed/graphic-7.gif [5]: /embed/graphic-10.gif