Abstract
Background SARS-CoV-2 antigen-detection rapid diagnostic tests (Ag-RDT) offer the ability to diagnose COVID-19 rapidly and at low cost; however, lower sensitivity has limited adoption of Ag-RDT in clinical settings.
Methods We compared Ag-RDT, nucleic acid amplification tests (NAAT), and clinical judgment alone for diagnosing COVID-19 among symptomatic patients. We investigated two scenarios: a high-prevalence hospital setting with 24-hour NAAT turnaround, and a lower-prevalence outpatient setting with 3-day NAAT turnaround. We simulated transmission from cases and contacts and relationships between time, viral burden, transmission, and case detection. We used decision curve analysis to compare diagnostic approaches, estimating the time- and infectivity-dependent benefit of each true-positive diagnosis.
Results In the primary analysis comparing Ag-RDT and NAAT, greater net benefit was achieved with Ag-RDT in the outpatient setting and with NAAT in the hospital setting. In the hospital setting, Ag-RDT becomes more beneficial if NAAT turnaround times exceed 2 days or Ag-RDT sensitivity increases to at least 95% (relative to NAAT) during acute illness. Similarly, in the outpatient setting, NAAT could be more beneficial when NAAT turnaround time remains under 2 days or patients strictly isolate while awaiting results. Clinical judgment was preferred only if clinical diagnoses generated a robust clinical and public health response and false-positive diagnoses produced minimal harm.
Conclusions For diagnosing symptomatic COVID-19, Ag-RDT may provide greater net benefit than either NAAT or clinical judgment when NAAT turnaround times are more than two days. NAAT is likely to remain optimal for hospitalized patients with prolonged symptoms prior to admission.
Introduction
Accurate diagnosis of COVID-19 can help to guide clinical management, interrupt transmission, and appropriately allocate resources to isolation, contact tracing, and treatment, while also minimizing the harms of misdiagnosis. In many settings – including low- and middle-income countries (LMICs) as well as high-income countries with large outbreaks – efforts to diagnose COVID-19 using highly sensitive molecular tests have quickly exceeded resource constraints or laboratory capacity.
Antigen-detection rapid diagnostic tests (Ag-RDTs) avoid the need for centralized laboratory processing and can be performed in less than an hour at lower cost than nucleic acid amplification testing (NAAT). Thus, they could facilitate higher volumes of testing and provide rapid results. A potential downside of Ag-RDTs, however, is their lower sensitivity relative to NAAT. Some experts have argued that all SARS-CoV-2 testing must be highly sensitive [1,2], while others advocate less sensitive testing primarily for community-level surveillance [3]. In many settings, however, the need for diagnostic testing in clinical settings outstrips NAAT capacity. Recognizing this, WHO and national public health agencies have issued target product profiles for Ag-RDTs [4] and interim guidance for their use in select circumstances [5–7]. Meanwhile, a global partnership has begun to make 120 million Ag-RDTs available for clinical diagnostic testing in LMICs [8].
There is a need to better understand under what conditions implementing Ag-RDTs would be preferable to reliance on NAAT and/or clinician judgment alone. This comparison requires consideration beyond accuracy (i.e., sensitivity and specificity). We therefore performed an analysis rooted in decision curve analysis (DCA), an approach commonly used to weigh risks of over- and under-treatment [9–11]. DCA combines the benefits of true-positive diagnoses and the harms of false-positive diagnoses on a single scale of “net benefit,” accounting for the prevalence of disease and the strength of preference for a true-positive versus false-positive diagnosis. We modified this approach to account for the importance of not only making correct diagnoses but also making them rapidly, on individuals with the highest transmission potential, and with an approach that is sufficiently specific to enable decisive action.
Methods
Diagnostic approaches and clinical populations modeled
We modeled three different approaches to diagnosing symptomatic individuals with COVID-19: point-of-care Ag-RDT, centralized NAAT, or clinical diagnosis alone. In the clinical diagnosis approach, interventions such as isolation, treatment, and/or contact notification and quarantine could be given to all patients in whom a COVID-19 diagnosis was considered (“treat all”) or to those whom clinicians deemed most likely to have COVID-19 (“clinician judgment”). For directness of comparison, we assumed that only a single diagnostic approach would be used in a given setting and patient population, with no further testing or clinical risk stratification after the initial evaluation.
We compared these three diagnostic approaches in each of two symptomatic patient populations: patients with moderate-to-severe illness in a hospital setting (of whom 40% are assumed to have COVID-19), and mildly symptomatic people in an outpatient setting (assuming that 10% have COVID-19 [12]). Besides prevalence, differences between the two settings included the turnaround time for NAAT (1 day in the hospital, 3 days in the outpatient setting), the extent to which patients were presumptively isolated while awaiting a diagnostic result (higher in the hospital setting, Table S1), and the clinical versus public health focus of interventions after a positive result.
Sensitivity, specificity, and net benefit estimation
Table 1 shows estimates of the sensitivity and specificity of Ag-RDT, NAAT and clinician judgment for detecting COVID-19 at the time of symptom onset. Table 1 also shows a simulated sensitivity among all patients presenting for clinical evaluation, after accounting for delayed presentations and declining viral burdens over time (further details below).
We used net benefit estimation to compare different diagnostic approaches on a single scale [9]. In this approach, the benefit of intervening on someone with the condition is defined relative to the harm of similarly intervening on someone without the condition. This relative benefit and harm can also be expressed as a “threshold probability” representing the degree of diagnostic certitude (i.e., probability that a patient has the condition of interest, in this case COVID-19) at which intervening and not intervening have equal expected benefit.
To combine the benefit gained by intervening on people with COVID-19 (“true positives”) and the harm of intervening on people without COVID-19 (“false positives”) on a single net benefit scale, we initially adopted the conventional approach of assuming that every true positive diagnosis provides the same expected benefit as every other. We then modified the net benefit calculation to capture certain relationships between the diagnostic approach and the value of a true-positive result. These relationships, summarized in Figure 1, include the greater value of COVID-19 diagnoses that are made rapidly, on patients with high virologic burden, and with greater certainty than a purely clinical diagnosis provides. The following section describes how these considerations were incorporated into estimates of net benefit.
Modeling relationships between diagnostic approach and benefit of case detection
Viral dynamics
We developed a stochastic model of SARS-CoV-2 transmission dynamics (described in greater detail in the Supplement) to simulate the temporal distribution of transmission events, the heterogeneity of peak viral burden and infectivity within the patient population, the timing of clinical presentation and diagnostic results, the dependence of assay detection on viral burden at the time of testing, and the resulting transmission avertable by a given diagnostic approach.
We simulated timing of transmission from cases, accounting for pre-symptomatic transmission, delays between symptom onset and clinical presentation, partial isolation while awaiting test results, and more stringent isolation and some degree of contact tracing once a COVID-19 diagnosis was made. In infected contacts, we simulated a distribution of incubation periods and of times to onward transmission. Thus, even with immediate intervention when a patient presented to care, some transmission from cases and from their contacts would have already occurred. Longer turn-around times for diagnostic results would further limit the potential for diagnosis to lead to reductions in transmission.
We assumed that the viral burden in patients’ diagnostic specimens at the time of symptom onset was log-normally distributed in the population [13,14], and that both diagnostic sensitivity and intensity of transmission were higher with increasing viral burden, such that cases who could be diagnosed only by a highly sensitive assay were less infectious, on average, that those who could be diagnosed by a less sensitive assay on the same day of illness. Based on observed relationships between NAAT cycle threshold, viral culture, and infectivity [15–18], we assumed that above a minimum threshold of 103 viral genome copies, the number of infections generated by an index case, in absence of intervention, was proportional to the log of his/her peak viral burden ([3]; see Supplement for details). We also assumed that declining viral burden led to declining assay sensitivity over time, with Ag-RDT sensitivity declining on a similar timeline as infectivity [19] and NAAT sensitivity declining more slowly [20] (details in Supplement).
Other variation in benefit of case diagnosis
Like transmission-related benefits, the clinical benefits of COVID-19 diagnosis are time sensitive. We modeled the avertible COVID-19 morbidity and mortality as decaying exponentially with time to diagnosis. Our primary analyses assume that 15% of the remaining potential clinical benefit of treating a true-positive diagnosis is lost with each day of delay (Supplement).
Finally, we consider that clinical diagnoses may result in less stringent isolation, less case reporting or contact notification, or more circumspect clinical management, compared to virological diagnoses made using Ag-RDT or NAAT. Therefore, sensitivity analyses consider the possibility that the benefits of a true-positive diagnosis and the harms of a false-positive diagnosis are reduced by a specified factor when the diagnosis is clinical rather than virological.
Net benefit calculations and decision curves
We present results as decision curves, which plot the net benefit of each diagnostic approach as a function of the threshold probability [11].
We first followed the conventional approach to estimating net benefit: Assuming all true-positives have equal weight, we tallied the benefits from true-positives and subtracted the harms of false-positives using a weight derived from the decision threshold probability. Specifically, the net benefit of assay j was calculated as where snj and spj are the sensitivity and specificity, respectively, of assay j among all patients evaluated (at the time of clinical presentation, not limited to those with acute illness), and q is the threshold probability (i.e., is the harm of a false positive relative to the benefit of a true positive).
For the modified net benefit approach, we first estimated the relative contributions of clinical and transmission-related benefit to the overall maximal benefit of a true-positive diagnosis in the hospital setting. This estimate (detailed in the Supplement) compared the relative mortality benefit of promptly treating a hospitalized COVID-19 patient to the benefit of preventing a SARS-CoV-2 infection (and thus having some probability of preventing a need for hospitalization). In the outpatient setting, the benefits of diagnosis were assumed to be exclusively transmission-related (thus excluding patients who would be eligible for treatments such as monoclonal antibody infusions). We then calculated the net benefit of diagnosis with assay j as where tj is the time from clinical presentation to diagnostic result with approach j; Rj is the proportion of transmission from the cases diagnosed by approach j or from their contacts that is averted with approach j (detailed in Supplement); c is the relative benefit of maximal improvement in clinical management, compared to the benefit of preventing all transmission from the same cases; d is the rate at which potential clinical benefit is lost as time passes; and lj is reduction in intensity of response associated with a given diagnostic method (i.e., lj ≤ 1 when j is clinical diagnosis). Other parameters are defined as in equation 1.
Decision curves plot net benefit across a range of values of the threshold probability q. For one-way and two-way sensitivity analyses of other parameters, we hold q fixed at a value estimated by considering the willingness in current practice to isolate or quarantine individuals with specified probabilities of having or developing COVID-19 (Supplement).
Parameter estimates are described in Table S1. All analyses were performed using R version 4.0.2.
Results
Transmission-related benefits of diagnosis
The transmission effects of each assay, which motivated our modified approach to net benefit, are shown for the outpatient setting in Figure 2. As a result of the modeled relationships between viral burden, infectivity, virological detection, and time (Figure S1), Ag-RDT diagnosed 62% of COVID-19 cases, compared to 80% diagnosed by each of NAAT and clinical judgment. All three approaches have similar effects on transmission, with Ag-RDT slightly outperforming NAAT in the primary analysis.
By contrast, among hospitalized patients, NAAT prevented a greater proportion of transmission than Ag-RDT, due to the assumptions of a shorter NAAT turnaround time and a stricter adherence to isolation while awaiting test results, compared to the outpatient setting (Figures S3, S4).
Conventional net benefit and decision curves
We first evaluated conventional decision curves, ignoring viral dynamics and time-dependence and assigning the same net benefit to every true-positive diagnosis. This approach suggested that NAAT, with its superior sensitivity and specificity, would be preferred over Ag-RDT at all values of the threshold probability (Figure 3, red and blue lines).
It also suggested that when the harms of overdiagnosis were judged to be low (threshold probabilities <21% in the hospital, <5% in the outpatient setting), clinical diagnosis could be superior to Ag-RDT. Under such circumstances, intervening on all patients suspected of having COVID-19 (the “treat-all” approach) would be the optimal clinical diagnosis strategy (Figure 3, solid black line).
Modified net benefit and resulting decision curves, Ag-RDT versus NAAT
Incorporating the time-dependent and viral load-dependent value of diagnosis into the analysis revealed more favorable profiles for the net benefit of Ag-RDT.
In the outpatient setting and at threshold probabilities below 20% (i.e. when the harm of missing a case was more than four times as great as the harm of a misdiagnosis, as often assumed in practice [supplement]), Ag-RDT was estimated to provide greater net benefit than NAAT (Figure 4A). NAAT could, however, outperform Ag-RDT if NAAT turnaround time were reduced to ≤2 days (Figure 4A), sensitivity and/or specificity of Ag-RDT were very low, adherence to isolation was good while awaiting a test result, or viral burden and infectivity were uncorrelated (Figure 5). NAAT could also outperform Ag-RDT in situations where the harm of intervening on a patient without COVID-19 was as great as the benefit of intervening on a patient with COVID-19 (threshold probabilities ≥50%, Figure 4A) – as might occur, for example, for high-risk patients who would be placed into congregate isolation facilities if they tested positive.
In the hospital setting, NAAT offered greater net benefit than Ag-RDT, although the difference was smaller than in the conventional net benefit model. In sensitivity analyses, NAAT lost its advantage over Ag-RDT if NAAT turnaround time increased from 1 day to 2 days and/or Ag-RDT sensitivity for detecting acute illness increased from 85% to >95% (Figure 4C and Figure 5).
As seen previously in the conventional net benefit analysis, clinical diagnostic approach could provide greater benefit than either Ag-RDT or NAAT testing if the relative harm of false-positive diagnoses was low (e.g., threshold of <2% in the outpatient setting or <15% in the hospital setting for preferring the treat-all approach, Figure 4B and 4D). In comparison, we estimated that an empirical threshold probability for COVID-19 diagnosis is around 10% but variable and highly context dependent (Supplement). Assuming that clinically diagnosed cases receive all the same interventions as virologically dianogised cases may also overestimate the benefit of a clinical diagnostic approach at low threshold probabilities. When we considered the possibility that this low diagnostic certainty could result in less intensive intervention (reducing the benefit of each true-positive diagnosis by 25% when made clinically), virological testing became the preferred diagnostic approach at nearly all threshold probabilities (Figure 4B and 4D, dotted lines).
Discussion
As antigen-detection SARS-CoV-2 rapid diagnostic tests become more widely available, it is important to identify the settings under which such tests offer incremental benefit to NAAT or clinical judgment alone. Using a novel adaptation of net benefit analysis, we demonstrate that Ag-RDT is likely to outperform NAAT under typical outpatient conditions that include several-day turnaround times for NAAT, even if the Ag-RDT has considerably lower sensitivity. We also demonstrate conditions under which Ag-RDT would be favored over NAAT in a hospital setting, including NAAT turnaround-times of more than 48 hours or Ag-RDT assays that achieve sensitivity >95% relative to NAAT during acute illness. We also demonstrate that, in absence of NAAT, Ag-RDT testing is preferable over clinical judgment-based or “treat-all” approaches under reasonable assumptions about the extent and effectiveness of interventions for virologically unconfirmed diagnoses.
Our novel application of decision curve analysis to COVID-19 demonstrates the importance of accounting for factors beyond sensitivity and specificity when using this tool to evaluate infectious disease diagnostics. Although NAAT offers higher sensitivity (and, in symptomatic patients, potentially better specificity) than Ag-RDT, the fact that Ag-RDT delivers results more rapidly and identifies the most highly infectious individuals can make it equivalent or superior to NAAT in averting transmission. Incorporating these considerations, we conclude that Ag-RDT would produce greater net benefit than NAAT in many clinical settings, when conventional decision curves based only on sensitivity, specificity, and threshold probability would suggest the opposite.
These estimates are likely to be conservative regarding the benefit of adopting Ag-RDT in clinical settings. First, our analysis does not account for the cost of diagnostic tests. Second, our primary estimates of Ag-RDT sensitivity in the acute phase (i.e., 85% relative to NAAT) may be conservative [18]. Third, our model does not account for the potential that patients who test falsely negative by Ag-RDT may still receive some degree of isolation or clinical intervention on the basis of high clinical suspicion. On the other hand, our model does not account for the costs of diagnostic uncertainty, in such forms as additional diagnostic workup or unnecessary treatment of true negatives, that may result from less definitive post-test probabilities after Ag-RDT compared to NAAT.
Our model is limited by underlying data availability about SARS-CoV-2 dynamics. In particular, data on the relationship between viral burden and infectivity remain sparse. Our ability to draw conclusions that apply across settings and assays is also limited by varying clinical and public health practice – for instance, the extent to which symptomatic people self-isolate or contact tracing is performed. Estimates of threshold probability are also context specific, depending, for example, on the extent of dependence on population-wide versus case-targeted interventions and on the individual and societal economic costs of those interventions in a particular setting. Third, certain parameter estimates – such as the importance of preventing downstream transmission relative to preventing poor clinical outcomes in people currently infected – depend on the future course of the epidemic more broadly. Preventing transmission may be more important in settings with emerging or widespread transmission and less important in settings with resolving epidemics or imminent widespread vaccination. Finally, our analysis focuses on diagnosis of symptomatic individuals and quarantine of their direct contacts, but Ag-RDT may also play an important role in preventing presymptomatic and asymptomatic transmission.
In conclusion, our modified decision curve analysis demonstrates that for symptomatic individuals, a moderately sensitive Ag-RDT could offer greater net benefit than either NAAT testing or clinician-driven diagnosis for COVID-19. This is true particularly in outpatient settings – where NAAT turnaround times are often long, prevalence relatively low, and the benefits of diagnosis dominated by the potential to reduce onward transmission. Ag-RDTs can also offer greater benefit than NAAT-based testing in hospitalized populations when NAAT delays are long or Ag-RDT sensitivity surpasses 95% that of NAAT. Among certain patients – those presenting with severe, treatment-eligible illness after a week or more of symptoms – NAAT test should be offered whenever possible, but for clinical use more broadly, countries and health systems where the need for clinical diagnostic testing outstrips capacity for completing NAAT within 48 hours should move rapidly to implement Ag-RDT in clinical settings.
Data Availability
Primary data used in this analysis is publicly available and cited in the manuscript. Model code is available from the authors upon request.
Funding
This work was supported by Foundation for Innovative New Diagnostics (FIND), through funding from the World Health Organization; and by National Institutes of Health [grant numbers NIH U54EB007958-12, U5411090366, and 3U54HL143541-02S2 to Y.C.M.).
Potential conflicts of interest
EAK, NA, YM, DWD declare no conflicts.
JAS, SD, and SGS declare that they are employed by the Foundation for Innovative New Diagnostics (FIND).