Validation testing to determine the effectiveness of lateral flow testing for asymptomatic SARS-CoV-2 detection in low prevalence settings

Lateral Flow device testing of students

As part of the national plan to test students for SARS-CoV-2 before the 2020 Christmas break, the DHSC provided University of Birmingham with Innova Lateral Flow Devices to test 15,000 students. Using a University owned booking system a total of 7,189 students were tested between December 2^nd and December 9^th. Students were provided with a sterile nasopharyngeal swab and under supervision from a trained member of the University testing team, swabbed both tonsils and a single nasal cavity. The swab was passed through an opening in a plastic screen to a designated test area, where it was immediately processed according to the Innova protocol. Tests were performed by trained members of the University testing team drawn from post-graduate and final year undergraduate students in the College of Medical and Dental Science in the University, supervised by highly experienced post-doctoral researchers. In total a team, of 18 test operatives oversaw 36 testing booths, with a student attending a booth every ten minutes. A further 7 staff acted as results recorders logging the test results via a barcode through a DHSC provided mobile phone device and result recording app.

Validation of Lateral Flow Device test results by PCR testing

University of Birmingham is home to a national pillar 2 testing laboratory, termed Turnkey lab, which conducts SARS-CoV-2 PCR diagnostics on behalf of DHSC (6). The laboratory uses the ThermoFisher Covid-19 taqPath assay used routinely in the Lighthouse laboratory testing network and tests a range of samples from mobile and stationary test sites (6). On each day of testing 90 residual Lateral Flow device test samples (saline solution in which the nasopharyngeal swab is resuspended to perform the test) were selected for confirmatory PCR testing. All positive samples were chosen for confirmatory PCR and the remainder were randomly selected samples. All samples were completely anonymous to the testing team with no identifying labels and were arbitrarily numbered from 1-90 each day. Sterile water was added to the samples to bring them to the 500 microlitres required for automated RNA extraction, and tested according to Pillar 2 laboratory protocol (6).

Statistical analysis of PCR validation

The efficient stratified study design involved verification of all Innova test positives with RT-PCR, and a random sample of 720 of the 7187 Innova test negatives. Estimates of sensitivity and prevalence with 95% confidence intervals were obtained using maximum likelihood inverse probability weighted logistic regression to account for the sampling design, with conversion of the estimated odds to probabilities. Weights of 1 for Innova test positives, and 9.98 (7187/720) for Innova test negatives were used. Expected numbers of cases were computed from the prevalence estimate. The estimate of specificity was obtained without weighting as no Innova test positives were observed in those with negative PCR. Exact binomial methods were used to compute confidence intervals for test yield and specificity.

Theoretical validation of LFD performance against Pillar 2 PCR test data

As part of pillar 2 testing our Turnkey laboratory also conducts PCR testing as part of the national Care Home surveillance plan implemented by DHSC to test all care home staff and residents to assist in control of Covid-19 transmission in UK care homes (7). Between October 25^th and November 5^th the Birmingham Turnkey laboratory processed a total of 19,176 PCR tests on home test and care home samples from across the United Kingdom. Of these 641 samples tested positive for SARS-CoV-2 using the cut off of two of three gene targets amplifying at a Ct value of 35 or under (6). This gives a positivity rate of 3.3%, around the rate that might be reasonably be expected in a large random surveillance of the UK population at that moment in time.

Validation of Pillar 2 PCR positive samples on the Lateral Flow Device

We randomly selected ten anonymous samples from the Turnkey laboratory returning PCR positive results, encompassing samples positive at Ct values either side of the 29-30 detection cutoff. Two drops of sample were added using the sample tube provided in the Innova test kit, and results measured after 25 minutes. Additionally we tested 36 samples containing the very recently described new variant of SARS-CoV-2 identified in South East England, in which the S gene product does not amplify in our Thermo Taq-path assay.

Lateral flow testing of University of Birmingham students

A total of 7,189 students voluntarily attended the asymptomatic student testing centre between December 2^nd and December 9^th. Students were refused a test if they had any symptoms of COVID-19 and were referred to a local test site for PCR testing. Results of four samples were void, and two samples tested positive for SARS-CoV-2 by lateral flow, a prevalence of 0.03% (95% CI 0.02% to 0.10%) in the students volunteering for a test (Table 1).

Lateral flow results validation by Pillar 2 PCR

The two samples positive by Lateral Flow, and another 710 randomly selected negative samples were transported to the University Turnkey laboratory for PCR testing (9.9% of sample total). Of the 712 samples tested by PCR, 8 were positive: The two positive by Lateral Flow and a further 6 samples negative on the Lateral Flow Device (table 1).

Our PCR validation data suggests a true prevalence rate in the student population tested of 0.86% (95% CI 0.40% to 1.86%). The overall sensitivity of the test in the tested student population was observed to be 3.23% (95% CI 0.60% to 15.59%). We estimate that there would have been 62 cases in the 7185 students, of which 60 were missed. There were no false positive results, observed specificity was 100% (95% CI: 99.48% to 100.00%)

We further investigated the PCR testing discrepancy by extracting the Ct values for the amplification curves for the 8 PCR positive samples (Table 2). Our data shows that the six samples testing false Negative by Lateral Flow all had Ct values > 29, whilst the two true positive samples had Ct values of 20 and 25. We collated the RT-PR raw data from three technical replicates of assays performed on the Qnostics SARS-CoV-2 analytical Q-panel – 01 and generated average Ct values for each of the known viral titres provided in the panel (Table 3). Using this data we determined that at 100 viral copies per ml (the designated LoD for the Innova LFD – 3) the equivalent Ct values for the pillar 2 PCR assay would be a Ct of 30.8 based on the N gene target, roughly in line with our PCR validation data.

Sensitivity of Lateral Flow device in the student population by Ct value

From our data the Lateral Flow Device test yield is 2.8 per 10,000 (0.3 to 10 per 10,000). The sensitivity of the Lateral Flow device in the tested population differs greatly dependent on the viral titre of the person tested. At a PCR Ct value < 29 the sensitivity was 100% (95% CI 15.8 to 100). However, at a Ct <= 29 this dropped to 9.1% (1.03 to 49.1), and at Ct < 33 dropped again to 5.01% (0.78 to 32.14).

Extrapolation of Pillar 2 PCR data to theoretically evaluate Lateral Flow device performance

We collated the raw RT-PCR data for all 641 of our positive samples as of November 5^th and ranked them according the N gene Ct value (Supplementary table). We then plotted the distribution of Ct values for our 641 positive samples (Fig 1). Using the LoD of 100 pfu/ml, we determined that this would correlate with an N gene Ct value of 30.8 plus one other gene target amplifying at a Ct < 35. By applying this theoretical level of performance to the LFD we determine that 99 of our positive samples would not be able to be detected by the Innova LFD given that the Ct value of N gene is above 30.8. This equates to 15.44% of our true positive RT-PCR samples being missed by the LFD. This means that the theoretically the Innova LFD when compared to Pillar 2 samples from low-prevalence, asymptomatic population screening similar to student and care home surveillance, would successfully detect 84.56% of all infections.

• Download figure
• Open in new tab

Figure 1:

Graph plotting raw Ct values (Y-axis) for all 641 positive samples in the Birmingham Turnkey lab (Y-axis). Ct values for each of the targets (Orf1, N, S) are plotted, with a sample only called positive if at least 2 of the three targets have a Ct < 35. The red line indicates the N gene Ct value equating to 100 viral copies/ml, the previously determined LoD for the Innova LFD.