Diagnostic Efficacy of Rapid Antigen Testing for SARS-CoV-2: The COVid-19 AntiGen (COVAG) study

Setting and participants

This prospective study was conducted at the Corona Test Centre Stuttgart Cannstatter Wasen. Individuals scheduled for rRT-PCR testing of nasopharyngeal swabs were advised of the study orally and in writing. Participants had to be aged ≥18 years and capable of understanding the nature, significance, and implications of the study. Children and adolescents <18 years of age and patients obviously suffering from clinical conditions requiring emergency hospitalisation were excluded. All participants provided written and informed consent. The study was approved by Ethics Committee II (Mannheim) of the University of Heidelberg (reference number 2020-417MF) and the German Institute for Drugs and Medical Devices.

We recorded demographic characteristics, reasons for testing, medical history, major risk factors, acute symptoms, and vital signs, including heart rate, blood pressure, body temperature, and oxygen saturation (Table 1). In addition to collecting the nasopharyngeal swabs for rRT-PCR testing, we collected two completely independent swab specimens to run two commercially available and widely used Ag-RDTs from Abbott Diagnostics (Abbott Panbio TM COVID-19 Ag Rapid Test, Abbott Rapid Diagnostics Jena GmbH, Jena, Germany, www.abbott.com/poct) and Roche Diagnostics (Roche-SD Biosensor SARS-CoV-2 Rapid Antigen Test, identical to SD BIOSENSOR Standard Q COVID-19 Ag, www.sdbiosensor.com; Roche Diagnostics; Mannheim, Germany, www.roche.com). Hereafter, we refer to the tests as Abbott-RDT and Roche-RDT, respectively. We randomly assigned the study population to three sampling groups according to the time sequence of collecting the nasopharyngeal swabs (group 1: rRT-PCR, RDT-Roche, RDT-Abbott; group 2: RDT-Roche, RDT-Abbott, rRT-PCR; and group 3: RDT-Abbott, rRT-PCR, RDT-Roche).

Blood samples (two 7·5 ml S-Monovette^® Serum-Gel tubes and a 2·7 ml S-Monovette^® K₃-EDTA tube, Sarstedt, Nümbrecht, Germany) were collected from the participants in a resting position to create a biobank. These backup samples were stored at -80 C for future experiments. The full study protocol is available through https://www.synlab.de/human/news-artikel/veroeffentlichung-des-protokolls-der-covid-19-antigen-studie-1398.

Analytical procedures

Both the Abbott-RDT and the Roche-RDT were carried out by medically educated staff according to the manufacturers’ instructions on-site at the Corona Test Centre Cannstatter Wasen, Stuttgart, Germany, immediately after sampling the nasopharyngeal swabs. The nasopharyngeal swabs for rRT-PCR testing were placed in 2 ml of a phosphate-buffered saline solution (ISOTON™ II Diluent, Becton Dickinson, Galway, Ireland) and delivered to the SYNLAB Medical Care Centre Leinfelden-Echterdingen. This ensured that the performers of the RDTs were unaware of the rRT-PCR-results.

SARS-CoV-2 RNA was extracted from the nasopharyngeal swab samples and purified using the PurePrep Pathogens kit and a PurePrep 96 instrument (Molgen, Veenendaal, the Netherlands). The rRT-PCR assay was performed using the RIDA^®GENE SARS-CoV-2 test kit (R-Biopharm, Darmstadt, Germany) on the CFX96 Touch Real-Time PCR detection device (Bio-Rad, Feldkirchen, Germany) according to the manufacturers’ instructions. This test kit targets the SARS-CoV-2 2 envelope (E) gene; samples producing a cycle threshold (Ct) ≥ 35 were considered positive by rRT-PCR.

We screened rRT-PCR-positive samples for SARS-CoV-2 variants of concern (VOC) B.1.1.7 (Alpha, United Kingdom), B.1.351 (Beta, South Africa), and P.1 (Gamma, Brazil) using VirSNiP SARS-CoV-2 Spike N501Y and VirSNiP SARS-CoV-2 Spike del H69/V70 from TIB Molbiol (Berlin, Germany) according to the supplier’s instructions. Genotyping was restricted to samples with a Ct ≥ 30 in rRT-PCR (260 of 338 samples). Samples with positive results for both the N501Y substitution and H69/V70 deletion were assigned to the Alpha variant. The presence of N501Y and absence of H69/V70 was considered the Beta or Gamma variant. Samples in which both N501Y and H69/V70 were absent may have contained variants other than Alpha, Beta, or Gamma or the wild-type. For the purpose of this article, these samples were termed negative for investigated mutations (NIM).

Statistical analysis

Continuous data are presented as the mean, standard deviation (SD), median, and 25^th and 75^th percentiles. Categorical data are presented as absolute numbers and percentages. Differences in continuous variables between men and women were examined by two-sided t-tests; the chi-squared test was used for categorical variables (Table 1). The risk of having COVID-19 according to baseline anthropometric and anamnestic characteristics was expressed in terms of crude odds ratios (ORs) and ORs adjusted for age and sex as calculated by logistic regression (Table 2). Sensitivity, specificity, positive predictive values (PPVs), negative predictive values (NPVs), and diagnostic efficacy of the two RDTs compared to rRT-PCR were calculated (Table 3). These performance indicators were compared between the Abbott-RDT and Roche-RDT. The P-value refers to two-sided testing of the null hypothesis, that the difference between the respective performance indicators is equal to zero and based on 5000 bootstrap iterations and subsequent percentile analysis. We also visualised the sensitivities of both RDTs relative to the rRT-PCR-derived Ct values (Figure 1) and the PPVs and NPVs according to hypothetical disease prevalence rates in the range of 0-0·05 (Figure 2). Finally, we investigated whether the sensitivities of the two RDTs were related to the reason for testing, comorbidities, clinical symptoms, vital signs, or SARS-CoV-2 genotypes using univariate (Table 3) and multivariate logistic regression (Table 4).

• Download figure
• Open in new tab

• Download figure
• Open in new tab

Figure 1.

Cycle threshold (Ct) values in rRT-PCR for SARS-Cov-2 RNA versus the sensitivities of the Roche-RDT (left) and Abbott-RDT (right). A. The solid lines indicate sensitivities, the dotted lines represent the upper and lower bounds of the corresponding 95% confidence intervals. B. The solid lines indicate sensitivities, the dotted lines represent the upper and lower bounds of the corresponding 95% confidence intervals. Red: SARS-CoV-2 NIM genotype; blue: SARS-CoV-2 Alpha variant.

• Download figure
• Open in new tab

Figure 2.

Positive predictive values (PPVs, right panel) and negative predictive values (NPVs, left panel) of two commercial RDTs for SARS-CoV-2-associated antigens in relation to disease prevalence rates up to 0·05. Red line: Abbott-RDT; blue line: Roche-RDT.

The statistical tests were two-sided and P<0·05 was considered significant. The analyses were carried out using R v4.0.2 (http://www.r-project.org).

Role of the funding source

The costs of the study were defrayed by SYNLAB Holding Deutschland GmbH. The management had no role in writing of the report or the decision to submit for publication. There was no financial support to SYNLAB Holding Deutschland GmbH from the manufacturers of the assays used in this evaluation and there has been no other financial support for this work that could have influenced its outcome. All authors had full access to all data and assume final responsibility for the decision to submit for publication.

Clinical characteristics of participants

The study was conducted between February 1, 2021, and March 31, 2021. During this period, nearly 17 000 adult persons attended the test centre Cannstatter Wasen to receive an rRT-PCR test. A total of 2222 persons agreed to participate in the study. Seven of them were disregarded for further evaluation because at least one of the three tests was not available. This resulted in 2215 persons with valid data (Table 1). Adverse events from performing any of the tests were not experienced.

We divided the reasons for testing into three major categories (Table 1): 707 (32%) participants were referred by their primary care physicians, 962 (43%) sought testing following the advice of the Health Department and were mostly incriminated contact persons of COVID-19 patients, and 546 (25%) participants were tested for miscellaneous reasons (264 following a warning message from the German Corona App, 140 out-of-pocket payers, 82 kindergartners or teachers, and 60 for another reason, including 2 for confirmation of a positive RDT and 2 cluster students in quarantine).

Hypertension, dyslipoproteinaemia, and diabetes mellitus were self-reported at rates of 12%, 5%, and 2%, respectively. Chronic obstructive lung disease and ischaemic heart disease were comparatively low in frequency. Overall, comorbidities occurred more often in men than in women (Table 1).

The most often reported clinical symptoms were malaise, headache, and musculoskeletal pain (Table 1). Symptoms were significantly more frequent in women than in men. Systolic and diastolic blood pressures were markedly and significantly higher in men than in women, but body temperature was slightly higher in women than in men (Table 1).

Risk of rRT-PCR-proven SARS-CoV-2 infection according to baseline characteristics

Among the 2215 participants, 338 carried SARS-CoV-2 based on rRT-PCR. Age and sex were not related to the likelihood of testing positive for SARS-CoV-2 (Table 2). Participants referred by treating physicians and health departments were positive significantly more often than participants with miscellaneous reasons for testing (OR 0·22, 95% confidence interval [CI] 0·14-0·34, adjusted for age and sex and compared to referrals from physicians).

Persons with at least one comorbidity more frequently tested positive than those without comorbidity (OR 2·94, 95% CI 2·25-3·83, adjusted for age and sex). Among the individual comorbidities, dyslipoproteinaemia, diabetes mellitus, and ischaemic heart disease significantly increased the probability of a positive rRT-PCR test. The presence of at least one clinical symptom at presentation resulted in a higher frequency of positive rRT-PCR (OR 2·91, 95% CI 2·28-3·71, adjusted for age and sex). In addition, each of the individual symptoms was positively and significantly related to the rate of SARS-CoV-2 detected by rRT-PCR. Among the objectively measured vital signs (blood pressure, body temperature, and oxygen saturation), elevated body temperature was associated with the probability of COVID-19.

Diagnostic performance of RDTs

Prevalence of SARS-CoV-2 infections in the study cohort

Among the attendees of our corona test centre, the rate of individuals testing positive for SARS-CoV-2 infection by rRT-PCR was 15%, which markedly exceeds the prevalence rate in the German population during the study (approximately 0·3% estimated on the basis of 7-day incidence rates during the study period). The probability of testing positive by rRT-PCR was not related to age and sex. This is in contrast to the expected over-representation of older people and may reflect pre-selection for persons having an evident clinical indication for testing. The probability of testing positive by rRT-PCR was strongly linked to individual reasons for testing, to the presence of comorbidities, clinical complaints, and elevated body temperature. For example, the persons referred by physicians due to suspected COVID-19 or those coming from the Health Department were more often positive than the group with miscellaneous reasons (kindergartners/teachers, out-of-pocket payers). Yet, in the latter group (prevalence rate 0·044), the proportion of SARS-CoV-2 carriers was still 10-fold higher than in the general population (assumed prevalence 0·003).

Sensitivity of the RDTs

Expectedly, both RDTs had higher sensitivities in subgroups with high viral loads (referral by physicians and health departments, clinical symptoms). However, both sensitivities and viral loads were low in patients presenting with comorbidities. This is unexpected and may reflect a referral bias in the sense that the indication for testing is more frequent and earlier in patients at high risk for severe COVID-19.

The relationship between the RDTs’ analytical sensitivity and viral load is in line with reports of sensitivities between 24·3% and 50% for RT-PCR-positive samples, which increased up to 81·8% and 100% for samples with high viral loads (>6 log₁₀ RNA copies/ml) ^8-10. In specimens from the upper respiratory tract, SARS-CoV-2 RNA peaks with the beginning of symptoms around day 4, decreases steadily during the first 10 days after illness onset, and can be detected up to 20 days after the onset of symptoms ^12-15. However, viral loads are low during the earlier stage of infection and in the second week after the onset of symptoms ^16,17. Thus, the time window for the detection of SARS-CoV-2 by RDTs appears to be narrower than with rRT-PCR and may be confined to the acute phase of infection. Consistently, we found a lower sensitivity of RDTs in asymptomatic individuals, suggesting limited usefulness of RDTs for screening such individuals, even if it is repeated on a regular basis.

We want to emphasise that the RDTs were carried out by medically educated personnel with strict adherence to the instructions issued by the manufacturers, perhaps explaining the low rate of false-positive results. Yet, RDTs have been widely recommended for self-testing or for testing by lay persons. Indeed, when Ag-RDT self-testing results were evaluated in a comparative study among symptomatic outpatients, self-testing (including self-read-out) yielded a sensitivity of 82·5% compared to professional nasopharyngeal sampling and testing ¹⁸. The same study noted variations in the sensitivity of Ag-RDT self-testing depending on the viral load of the sample. High viral loads ≥7·0 log10 SARS-CoV-2 RNA copies/ml led to a sensitivity of 96·6%, whereas low viral loads <7·0 log10 SARS-CoV-2 RNA copies/ml had decreased sensitivity (45·6% for Ag-RDT self-use and 54·5% for Ag-RDT professional use) ¹⁸. Thus, it appears that, at low viral loads, as encountered in asymptomatic persons, self-administered RDT testing may be even less effective in reality than it was in the current study.

Sensitivity for the SARS-CoV-2 Alpha variant

During the conduct of our study, the SARS-CoV-2 Alpha variant became the prevailing genotype in Southern Germany. Consequently, the study included 83 carriers of SARS-CoV-2 NIM, 166 carriers of the Alpha variant, and 4 patients with other viral genotypes (251 of 338 samples with Ct values ≥ 30). Remarkably, the sensitivities of both RDTs were lower in Alpha variant carriers, and this finding was robust against adjustments for the viral load expressed in terms of Ct values (Table 4). The Alpha variant may be differentiated from the wild-type by two key mutations in the Spike protein: the N501Y substitution within the receptor-binding domain and the H69/V70 deletion. It may be approximately 80% more transmissible than the wild-type ¹⁹ due to conformational changes increasing the Spike protein’s affinity for the angiotensin-converting enzyme 2 (ACE2) receptor ^20-22. SARS-CoV-2 variants may also confer decreased binding of therapeutic antibodies and protection by vaccination, whereby the Alpha variant may display the smallest variation in antigenicity compared to other circulating variants. This raises the attractive possibility that the lower reactivity of the current RDTs for the Alpha variant was related to structural alterations in the epitope(s) recognised by the detecting antibodies. However, the antibodies incorporated in both RDTs recognise the nucleocapsid protein (N-protein) ²³. In a laboratory-based investigation (virus suspended in cell culture medium and saliva), the performance of the Roche-RDT and Abbott-RDT was not affected by variants. This is in contrast to the current results, which were collected in a real-world setting. A potential explanation for the discrepancies may be that both the Spike protein and the N-protein tightly interact, and that conformational changes in the Spike protein may affect the three-dimensional structure and accessibility for antibodies against the N-protein ²⁴. Finally, we cannot rule out that the structural changes in other variants may be even greater than in the Alpha variant ²⁵, so that their effect on sensitivity may be even stronger. Thus, any validation of RDTs for SARS-CoV-2 would also have to be extended to known and future variants rather than limited to the wild-type SARS-CoV-2.

Implications for screening

A recent meta-analysis issued by the Cochrane Collaboration reviewed 48 studies including 58 commercial RDTs. It reported sensitivities between 34·1% (95% CI 29·7% - 38·8%) and 88·1% (95% CI 84·2% - 91·1%) for RDTs in symptomatic persons, whereas in asymptomatic persons, the sensitivities varied between 28·6%(8·4% to 58·1%) and 69·2% (38·6% to 90·9%) ¹¹.

The Cochrane analysis identified three studies on the Abbott-RDT with 1094 symptomatic participants, including 252 SARS-CoV-2 cases, and one study with 474 asymptomatic persons and 47 cases. It also identified three studies with 1948 symptomatic participants and 336 cases and one study with 127 asymptomatic persons and 13 cases for the Roche-RDT (listed as Biosensor Standard Q). The number of asymptomatic persons in the current study at least exceeds the number in the study in the Cochrane meta-analysis ¹¹.

For the Abbott-RDT, the Cochrane analysis reported sensitivities of 75·1% (57·3% to 87·1%) and 48·9% (35·1% to 62·9%), and specificities of 99·5% (98·7% to 99·8%) and 98·1% (96·3% to 99·1%) in symptomatic and asymptomatic persons, respectively. The sensitivity of the Roche-RDT was reported to be 88·1% (84·2% to 91·1%) and 69·2% (38·6% to 90·9%) in symptomatic and asymptomatic patients, respectively, with specificities similar to the Abbott-RDT. The current findings almost exactly coincide with the Cochrane analysis and significantly extends the available evidence.

Analogous with the viewpoint of the Cochrane analysis ¹¹, at a prevalence rate of 0·05, sensitivities of 60·4% and 56·8% and specificities of 99·7% and 99·9% with the Roche-RDT and Abbott-RDT, respectively, in symptomatic patients with SARS-CoV-2 infections, our data would translate as follows.

If 10 000 patients and 500 (0·05) true positives were examined, 414 and 397 persons would have tested positive, of which 38 and 25 would have been false-positives and 124 and 128 persons with negative test results would be falsely negative for Roche-RDT and Abbott-RDT, respectively (Table 5). Assuming a prevalence rate of 50 (0·005) in 10 000 asymptomatic patients, the respective figures would be 33 and 16 persons testing positive, of which 17 and 4 would have been false-positives for Roche-RDT and Abbott-RDT, respectively, and 34 and 38 persons with a negative test would be falsely negative. This is crucially important, as RDTs have specifically been recommended for screening asymptomatic persons (Table 5). Furthermore, as shown in Table 5, these figures would be substantially affected if the SARS-CoV-2 Alpha variant is predominant.

Limitations

We applied rRT-PCR as the reference method to detect SARS-CoV-2 infection. Despite being considered the gold standard, this technique has the limitation that the detection of SARS-CoV-2 RNA in patient samples does not indicate the presence or shedding of viable virus with replicative capacity or whether the tested individual is contagious at the time of the test ^12,26. Although the presence of viral RNA proven by rRT-PCR does not automatically equate to infectiousness, a significant correlation between the Ct value (reflecting viral load) and subsequent virus cultivation has been observed. Samples with Ct values between 13 and 17 have a culture positivity rate of 100%, which declines gradually to 12% when Ct values of 33 are analysed. No viral growth occurs at Ct values ≥34, suggesting that patients with these values do not excrete infectious viral particles ¹⁷. Therefore, if infectivity rather than a positive rRT-PCR test were considered the reference for RDTs, our results would stand more in favour of RDT testing. However, a direct conversion of Ct values or a positive RDT to contagiousness has not yet been established. Ct values can hardly be compared across studies, and the correlation between viral load and the risk of transmission from a positive case is still not entirely clear ^13,27, with a variety of circumstances, such as the individual’s behaviour, the type and duration of contact, the environment, and the implementation of transmission-reducing measures (e.g., filter masks) affect infectiveness ^27,28.

The sensitivity of RDTs may relate to the time elapsed since infection. It is a limitation of the current study that the time point of infection in rRT-PCR-positive samples was not available. However, we consider the Ct values from rRT-PCR as a good proxy for the changes in viral load during the course of the study.

Finally, we demonstrated that the SARS-CoV-2 Alpha variant markedly diminishes the sensitivity of both RDTs. We cannot explain this finding. Other variants were not encountered in sufficient numbers, and we were not able to infer the sensitivities of the RDTs for other variants.

Directions for future research

This evaluation of the most sensitive RDTs currently available for SARS-CoV-2 suggests that screening asymptomatic persons with this approach may fail to identify a substantial proportion of viral carriers. Thus, further methodical refinements are needed, such as attempts to determine the viral load at least semi-quantitatively. Alternatively, rapid, on-site, direct detection of SARS-CoV-2 RNA RDTs could be pursued. The lower sensitivity of the RDTs for the Alpha variant indicates that their performance may be substantially influenced by the virus genotype, and strategies need to be developed to ensure that any of the circulating variants are captured. Finally, further head-to-head research is needed into how current screening strategies, RDT or laboratory-based, directly translate into controlling virus transmission and spread in the population. This will show whether the obvious practical advantage of RDTs is offset by their limitations.