Skip to main content
medRxiv
  • Home
  • About
  • Submit
  • ALERTS / RSS
Advanced Search

Prevalence Threshold and Temporal Interpretation of Screening Tests: The Example of the SARS-CoV-2 (COVID-19) Pandemic

Jacques Balayla, Ariane Lasry, Yaron Gil, Alexander Volodarsky-Perel
doi: https://doi.org/10.1101/2020.05.17.20104927
Jacques Balayla
1Department of Obstetrics and Gynecology, McGill University, Montreal, Quebec, Canada
MD, MPH
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • For correspondence: Jacques.balayla@mail.mcgill.ca
Ariane Lasry
1Department of Obstetrics and Gynecology, McGill University, Montreal, Quebec, Canada
B.Sc
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Yaron Gil
1Department of Obstetrics and Gynecology, McGill University, Montreal, Quebec, Canada
MD
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Alexander Volodarsky-Perel
1Department of Obstetrics and Gynecology, McGill University, Montreal, Quebec, Canada
MD
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • Abstract
  • Full Text
  • Info/History
  • Metrics
  • Data/Code
  • Preview PDF
Loading

Abstract

The curvilinear relationship between a screening test’s positive predictive value (PPV) and its target disease prevalence is proportional. In consequence, there is an inflection point of maximum curvature in the screening curve defined as a function of the sensitivity (a) and specificity (b) beyond which the rate of change of a test’s PPV declines sharply relative to disease prevalence (ϕ). Herein, we demonstrate a mathematical model exploring this phenomenon and define the prevalence threshold point (ϕe) where this change occurs as:

Embedded Image

Understanding where this prevalence point lies in the curve has important implications for the interpretation of test results, the administration of healthcare systems, the implementation of public health measures, and in cases of pandemics like SARS-CoV-2, the functioning of society at large. To illustrate the methods herein described, we provide the example of the screening strategies used in the SARS-CoV-2 (COVID-19) pandemic, and calculate the prevalence threshold statistic of different tests available today. This concept can help contextualize the validity of a screening test in real time, thereby enhancing our understanding of the dynamics of the current pandemic.

Introduction

The novel SARS-CoV-2 (COVID-19) has reached over 4 million confirmed cases worldwide; in the United States alone, the death toll totals close to 90,000 to date [1]. Despite efforts to contain its spread, the number of confirmed cases has continued to rise. These estimations have been largely based on polymerase chain reaction (PCR) detection of actively replicating viral material, found only among individuals who are actively infected at the time of testing. Those who recover from the virus without having been tested during the window of viral shedding are not included in prevalence estimates; neither are asymptomatic individuals, roughly approximated as 18 to 30% of those infected [2, 3]. Current estimates thus understate the true prevalence of COVID-19 by failing to include the aforementioned groups. Whereas accurate prevalence estimates are needed to inform public health measures, screening methods that depend on actively replicating viral organisms will become less reliable as recovery proceeds and the resulting prevalence of active infection among the population tapers. Bayes theorem thus demonstrates the point beyond which the positive predictive value (PPV) of currently employed PCR screening will decline as the curve ‘flattens’ over time. In contrast, serologic testing for COVID-19 antibodies persist in plasma well beyond the period of active infection; as recovery from active infection increases, so too will the cumulative prevalence of seropositivity among the population over time. Herein, we use differential equations to assess the geometry of screening curves and aim to describe the prevalence threshold point beyond which the PPV of various COVID-19 screening tests declines most acutely. Though this example is specific to the COVID-19 pandemic, the methods herein described apply to all screening tests whose sensitivity and specificity parameters are known.

Validity of Screening Tests

The validity of a screening test is defined as the ability to correctly delineate individuals who have a given disease or condition from those who do not. The following four parameters are used to assess the validity of screening tests: sensitivity a, specificity b, positive predictive value ϕ, and negative predictive value σ. Sensitivity refers to the proportion of people with a given disease who test positive for said disease, also termed the true positive rate. Specificity, also termed true negative rate, refers to the proportion of people without said disease who indeed test negative. Sensitivity and specificity are properties inherent to the screening test itself and are unaffected by the prevalence of disease in a given population. On the other hand, the positive predictive value is defined as the percentage of patients with a positive test that do in fact have the disease, and conversely, the negative predictive value refers to the percentage of patients with a negative test that do not have the disease. These two parameters are dependent upon the prevalence of disease being studied. Using Bayes’ theorem, we can derive the following equation expressing positive predictive value p(ϕ) as a function of disease prevalence ϕ.

Equation 1. The Screening Curve Equation Embedded Image where:

ϕ = prevalence, p(ϕ) = positive predictive value, a = sensitivity and b = specificity

Prevalence Threshold of a Screening Test

Bayes’ theorem describes the probability of an event, based on prior knowledge of conditions that might be related to the event. The relationship between a screening test’s positive predictive value, p(ϕ), and its target disease prevalence ϕ proportional - though not linear in all but one case where the sum of sensitivity and specificity equals one. From this curvilinear relationship, as stipulated in Figure 1, we can derive the prevalence threshold ϕe at which the sharp inflection point in the screening curve occurs, as depicted by the following equation (derivation available as a supplement).

Equation 2. The Prevalence Threshold Equation Embedded Image where:

ϕe = prevalence threshold, a = sensitivity and b = specificity

Nasopharyngeal Swab RT-PCR Screening Test for COVID-19

The Food and Drug Administration (FDA) published the analytical sensitivity and specificity of the currently employed COVID-19 reverse transcription polymerase chain reaction (RT-PCR) test [4], The limit of detection (LoD) for the minimum detectable concentration of SARS-CoV-2 genome (ie. sensitivity) in-vitro was set at 95%. Additionally, the COVID-19 RT-PCR was tested against 30 respiratory microorganisms, which yielded a specificity nearing 100%. While the above are analytical properties of the COVID-19 RT-PCR test, estimates of its clinical properties vary significantly between studies – particularly since some margin of error may result from improper sampling by health care workers (wrong angle, too fast to withdraw and swab, etc.), varying viral load and the subclinical stage when screening is carried out (colonization, incubation, prodrome or acute infection). We will therefore use the FDA’s published estimates for simplicity, where ϕe = prevalence threshold for detection of COVID-19, a = sensitivity = 0.95, b = specificity = 0.99 and a + b = 1.94. Thus, as per equation 2, the prevalence threshold for RT-PCR detection of COVID-19 is calculated as follows:

Equation 3. Prevalence threshold for the COVID-19 RT-PCR screening test

Embedded Image

In graphic form, the screening curve depicts the PPV as a function of prevalence as such:

Figure
  • Download figure
  • Open in new tab

The vertical line in red represents the prevalence threshold ϕe at 0.093 (9.3%).

In clinical practice, the sensitivity and specificity of the COVID-19 RT-PCR test are likely lower than those estimated analytically by the FDA; the prevalence threshold for detection that we have estimated above is thus likely conservative. Nonetheless, the implication is such that below a prevalence of 9.3% actively replicating cases of COVID-19 in the population, the PPV of RT-PCR testing declines almost exponentially. In other words, we would expect a sharp increase in the number of false positive screening tests, in turn falsely increasing the estimated prevalence of disease. Even with a conservative estimation of the prevalence threshold, there are potentially significant health, social, and economic implications of false positive screening tests.

Serology Testing as a Public Health Tool

Serology testing for SARS-CoV-2 involves blood-based testing for antibodies to COVID-19. This screening method identifies all groups of individuals sub-acutely infected or recovered from COVID-19, including those who may be asymptomatic at the time of testing. Use of this tool can thus provide public health officials with a more reliable estimation of the spread of COVID-19 and its cumulative prevalence among different populations over time. Furthermore, this information can bring forth attempts at a better understanding of disease transmission and immunity, which regarding COVID-19, is largely uncertain up until this point.

The Johns Hopkins Center for Health Security recently released a report outlining the sensitivity and specificity of various serology tests approved for diagnostic use in the United States [5]. These are summarized in Table 1, with prevalence thresholds estimated for each test. In contrast to RT-PCR screening, COVID-19 testing with serology can delineate immune individuals at a prevalence threshold as low as 4.3%. As the seropositive rates continue to increase, we deduce that above a prevalence of 4.3%, a positive COVID-19 IgG screen can be reliably accepted as a true positive. In contrast, the nasal RT-PCR test is most reliable when over 9.3% of the population has actively replicating virus at any given time – a value thankfully thus far not reached, even after accounting for an excess positive cases that are not tested.

View this table:
  • View inline
  • View popup
  • Download powerpoint
Table 1. Prevalence thresholds (ϕe) for different serologic tests available

Several countries have already started employing serology-based testing, either for research purposes or to grant ‘immunity certificates’ to those in whom antibodies to COVID-19 are found [6]. However, we must bear in mind that just as with RT-PCR testing, serologic testing has inherent limitations related to the lag between acute infection and the development of IgM and IgG antibodies as well. In the United States, the Centers for Disease Control and Prevention (CDC) recently released a COVID-19 Serology Surveillance Strategy [7]. This strategy aims to employ serology testing, termed ‘seroprevalence,’ at the large-scale (ie. highly impacted areas such as New York and Washington), the community-scale (with systematic selection of participants) and the small-scale (specific subgroups, eg. healthcare workers). From the reasoning above, we conclude that the best screening method to assess for acute infectivity, the need for isolation, and the ensuing social and economic repercussions, is nasal swab RT-PCR for actively replicating virus. In contrast, the most reliable screening method to assess for disease prevalence and burden of disease is antibody testing via serology – assuming enough time has lapsed since the acute infectious episode.

Note on Rapid Diagnostic Testing

In an effort to expedite testing speed and availability, rapid diagnostic tests (RDT) have been developed for point-of-care use [8-17]. These tests rely on serological markers against SARS-CoV-2, either IgM or IgG, for diagnosis of acute COVID-19 infection. While the FDA has approved various RDTs for use under Emergency Use Authorization, the WHO currently does not recommend their use outside of research purposes due to their inconsistent sensitivities [18]. Indeed, concerns over reports of false negative results prompted an FDA news release cautioning the public about the potential inaccuracy of Abbott’s ID NOW point-of-care RDT [13-17, 19]. Further studies with larger sample sizes and clear manufacturer instructions for correct sampling are needed in order to determine whether RDTs are reliable for clinical use.

Conclusion

The aforementioned values for sensitivity and specificity are likely to change as more samples are obtained. However, their values as stipulated here are used not to endorse a particular test, but to illustrate the concept of prevalence thresholds in the context of the current pandemic. To illustrate the methods herein described, we provide the example of the screening strategies used in the SARS-CoV-2 (COVID-19) pandemic, and calculate the prevalence threshold statistics of different tests available today. This concept can help contextualize the validity of a screening test in real time, thereby enhancing our understanding of the dynamics of the current pandemic. [Identifying where this prevalence point lies in the curve has important implications for the interpretation of test results, the administration of healthcare systems, the implementation of public health measures, and in cases of pandemics like SARS-CoV-2, the functioning of society at large.

Data Availability

We used publicly available estimates for sensitivities and specificities of different COVID-19 tests, as published by the Center for Health Security at John Hopkins University.

https://www.centerforhealthsecurity.org/resources/COVID-19/serology/Serology-based-tests-for-COVID-19.html

Footnotes

  • No funding was received for this study.

  • The authors have no conflicts of interest to report.

References

  1. 1.↵
    WHO Coronavirus Disease (COVID-19) Dashboard. May 17, 2020}; Available from: https://covid19.who.int.
  2. 2.↵
    Mizumoto, K., et al., Estimating the asymptomatic proportion of coronavirus disease 2019 (COVID-19) cases on board the Diamond Princess cruise ship, Yokohama, Japan, 2020. Euro Surveill, 2020. 25(10).
  3. 3.↵
    Nishiura, H., et al., Estimation of the asymptomatic ratio of novel coronavirus infections (COVID-19). Int J Infect Dis, 2020. 94: p. 154–155.
    OpenUrlCrossRefPubMed
  4. 4.↵
    ACCELERA TED EMERGENCY USE A UTHORIZA TION (EUA) SUMMARY COVID-19 RT-PCR TEST (LABORATORY CORPORATION OF AMERICA). U.S. Food and Drug Administration. p. 1–12.
  5. 5.↵
    Serology-based tests for COVID-19. May 17, 2020}; Available from: https://www.centerforhealthsecurity.org/resources/COVID-19/serology/Serology-based-tests-for-COVID-19.html.
  6. 6.↵
    Gigi Gronvall, N.C., Amanda Kobokovich, Rachel West, Kelsey Lane Warmbrod, Matthew P Shearer, Lucia Mullen, Tom Inglesby, Developing a National Strategy for Serology (Antibody Testing) in the United States. The Johns Hopkins Center for Health Security. p. 1–38.
  7. 7.↵
    COVID-19 Serology Surveillance Strategy. May 17, 2020}; Available from: https://www.cdc.gov/coronavirus/2019-ncov/covid-data/serology-surveillance/index.html.
  8. 8.↵
    DPP COVID-19 IgM/IgG System. 2020, ChemBio Diagnostic Systems. p. 1–31.
  9. 9.
    Cellex qSARS-CoV-2 IgG/IgMRapid Test. 2020. p. 1-5.
  10. 10.
    Pan, Y., et al., Serological immunochromatographic approach in diagnosis with SARS-CoV-2 infected COVID-19 patients. J Infect, 2020.
  11. 11.
    Li, Z., et al., Development and clinical application of a rapid IgM-IgG combined antibody test for SARS-CoV-2 infection diagnosis. J Med Virol, 2020.
  12. 12.
    Cassaniti, I., et al., Performance of VivaDiag COVID-19 IgM/IgG Rapid Test is inadequate for diagnosis of COVID-19 in acute patients referring to emergency room department. J Med Virol, 2020.
  13. 13.↵
    Zhen, W., et al., Clinical Evaluation of Three Sample-To-Answer Platforms for the Detection of SARS-CoV-2. J Clin Microbiol, 2020.
  14. 14.
    Smithgall, M.C., et al., Comparison of Cepheid Xpert Xpress and Abbott ID Now to Roche cobas for the Rapid Detection of SARS-CoV-2. J Clin Virol, 2020: p. 104428.
  15. 15.
    Rhoads, D.D., et al., Comparison of Abbott ID Now, Diasorin Simplexa, and CDC FDA EUA methods for the detection of SARS-CoV-2 from nasopharyngeal and nasal swabs from individuals diagnosed with COVID-19. J Clin Microbiol, 2020.
  16. 16.
    Harrington, A., et al., Comparison of Abbott ID Now and Abbott m2000 methods for the detection of SARS-CoV-2 from nasopharyngeal and nasal swabs from symptomatic patients. J Clin Microbiol, 2020.
  17. 17.↵
    Basu, A., et al., Performance of the rapid Nucleic Acid Amplification by Abbott ID NOW COVID-19 in nasopharyngeal swabs transported in viral media and dry nasal swabs, in a New York City academic institution. bioRxiv, 2020: p. 1–19.
  18. 18.↵
    Advice on the use of point-of-care immunodiagnostic tests for COVID-19: Scientific brief World Health Organization. p. 1–2.
  19. 19.↵
    Coronavirus (COVID-19) Update: FDA Informs Public About Possible Accuracy Concerns with Abbott ID NOW Point-of-Care Test. 2020, U.S. Food and Drug Administration.
Back to top
PreviousNext
Posted May 22, 2020.
Download PDF
Data/Code
Email

Thank you for your interest in spreading the word about medRxiv.

NOTE: Your email address is requested solely to identify you as the sender of this article.

Enter multiple addresses on separate lines or separate them with commas.
Prevalence Threshold and Temporal Interpretation of Screening Tests: The Example of the SARS-CoV-2 (COVID-19) Pandemic
(Your Name) has forwarded a page to you from medRxiv
(Your Name) thought you would like to see this page from the medRxiv website.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.
Share
Prevalence Threshold and Temporal Interpretation of Screening Tests: The Example of the SARS-CoV-2 (COVID-19) Pandemic
Jacques Balayla, Ariane Lasry, Yaron Gil, Alexander Volodarsky-Perel
medRxiv 2020.05.17.20104927; doi: https://doi.org/10.1101/2020.05.17.20104927
Reddit logo Twitter logo Facebook logo LinkedIn logo Mendeley logo
Citation Tools
Prevalence Threshold and Temporal Interpretation of Screening Tests: The Example of the SARS-CoV-2 (COVID-19) Pandemic
Jacques Balayla, Ariane Lasry, Yaron Gil, Alexander Volodarsky-Perel
medRxiv 2020.05.17.20104927; doi: https://doi.org/10.1101/2020.05.17.20104927

Citation Manager Formats

  • BibTeX
  • Bookends
  • EasyBib
  • EndNote (tagged)
  • EndNote 8 (xml)
  • Medlars
  • Mendeley
  • Papers
  • RefWorks Tagged
  • Ref Manager
  • RIS
  • Zotero
  • Tweet Widget
  • Facebook Like
  • Google Plus One

Subject Area

  • Epidemiology
Subject Areas
All Articles
  • Addiction Medicine (280)
  • Allergy and Immunology (579)
  • Anesthesia (139)
  • Cardiovascular Medicine (1944)
  • Dentistry and Oral Medicine (252)
  • Dermatology (184)
  • Emergency Medicine (333)
  • Endocrinology (including Diabetes Mellitus and Metabolic Disease) (698)
  • Epidemiology (11100)
  • Forensic Medicine (8)
  • Gastroenterology (623)
  • Genetic and Genomic Medicine (3164)
  • Geriatric Medicine (308)
  • Health Economics (561)
  • Health Informatics (2042)
  • Health Policy (863)
  • Health Systems and Quality Improvement (782)
  • Hematology (310)
  • HIV/AIDS (682)
  • Infectious Diseases (except HIV/AIDS) (12717)
  • Intensive Care and Critical Care Medicine (707)
  • Medical Education (317)
  • Medical Ethics (92)
  • Nephrology (334)
  • Neurology (2984)
  • Nursing (164)
  • Nutrition (463)
  • Obstetrics and Gynecology (589)
  • Occupational and Environmental Health (614)
  • Oncology (1550)
  • Ophthalmology (477)
  • Orthopedics (185)
  • Otolaryngology (266)
  • Pain Medicine (202)
  • Palliative Medicine (57)
  • Pathology (402)
  • Pediatrics (912)
  • Pharmacology and Therapeutics (381)
  • Primary Care Research (355)
  • Psychiatry and Clinical Psychology (2782)
  • Public and Global Health (5590)
  • Radiology and Imaging (1093)
  • Rehabilitation Medicine and Physical Therapy (632)
  • Respiratory Medicine (759)
  • Rheumatology (338)
  • Sexual and Reproductive Health (311)
  • Sports Medicine (289)
  • Surgery (343)
  • Toxicology (48)
  • Transplantation (159)
  • Urology (132)