Abstract
Molecular testing for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) is the gold standard for diagnosis of coronavirus disease 2019 (COVID-19), but the test clinical performance is poorly understood. From 3/10/2020-5/1/2020 NewYork-Presbyterian laboratories performed 27,377 SARS-CoV-2 molecular assays from 22,338 patients. Repeat testing was performed in 3,432 patients, of which 2,413 had negative and 1,019 had positive first day results. Repeat-tested patients were more likely to be older, male, African-American or Hispanic, and to have severe disease. Among the patients with initially negative results, 18.6% became positive upon repeat-testing. Only 58.1% of any-time positive patients had a result of “detected” on the first test. The clinical sensitivity of COVID-19 molecular assays is estimated between 66.2 % and 95.6%, depending on the unknown number of false negative results in single-tested patients. Conversion to a negative result is unlikely to occur before 15 to 20 days after initial testing or 20-30 days after the onset of symptoms, with 50% conversion occurring at 28 days after initial testing. Forty-nine initially-positive patients converted to negative and then back to positive in subsequent days. Conversion from first day negative to positive results increased linearly with each day of testing, reaching 25% probability in 20 days. In summary, our study provides estimates of the clinical performance of SARS-CoV-2 molecular assays and suggests time frames for appropriate repeat testing, namely 15 to 20 days after a positive test and the same or next 2 days after a negative test in a patient with high suspicion for COVID-19.
Introduction
Coronavirus disease 2019 (COVID-19) which is caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) (1, 2) is a global pandemic with mortality significantly higher than seasonal influenza (3). The rapid sequencing of SARS-CoV-2 genomes (1, 2) has allowed the development of multiple real-time reverse transcription-polymerase chain reaction (RT-PCR) assays that have become the gold standard to detect viral RNA and identify patients with COVID-19 as well as asymptomatic carriers. However, some patients with positive chest radiologic findings and symptoms suspicious for COVID-19 have been reported to test negatively by SARS-CoV-2 RT-PCR and require multiple consecutive tests to convert to a positive result (4, 5). There is limited information on the clinical performance characteristics of the SARS-CoV-2 molecular tests in real world cases, namely what is the predictive value of a negative result in patients suspected of COVID-19 and what is the relationship between the course of the disease, viral shedding, and positivity of the various molecular assays. During the 2020 pandemic of COVID-19, we tested 27,377 samples from 22,338 patients with SARS-CoV-2 molecular assays performed at NewYork-Presbyterian (NYP) clinical laboratories. The assays used in this patient population included six platforms based on reverse-transcription and nucleic acid amplification to detect SARS-CoV-2 specific RNA sequences. Our goal is to identify the clinical performance characteristics of SARS-CoV-2 molecular assays for diagnosis and stratification of COVID-19 patients and determine the dynamics of SARS-CoV-2 molecular assay results over time in a large dataset from multiple hospital locations in the New York City area.
Methods
Laboratory Dataset
We extracted records of all the SARS-CoV-2 tests performed at NYP laboratories from our laboratory information system (Cerner Millennium, Cerner Corporation, North Kansas City, MO) using a custom Cerner Command Language query. We tested a total number of 22,338 patients between March 10, 2020 and May 1, 2020 for SARS-CoV-2 by molecular assays at NewYork-Presbyterian affiliated hospitals and performed a total of 27,377 assays on nasopharyngeal (initially also on oropharyngeal) swab samples. The vast majority of the SARS-CoV-2 tests were reverse-transcription real time PCR assays performed with the high-throughput automated cobas 6800 (Roche Molecular Systems, Branchburg, NJ) platform (N=19,195), which went live at Columbia-NYP laboratory on 3/15/2020 and at Cornell-NYP laboratory on 3/30/2020. A smaller number of assays were performed since 3/10/2020 with in-house developed tests under FDA Emergency Use Authorization using the Rotor-Gene Q (N=1795, Qiagen, Valencia, CA) and 7500 Fast (N=89, Thermo-Fisher Scientific, Waltham, MA) instruments. From 4/3/2020 thereafter other platforms were introduced in the NYP laboratories, including the Xpert Xpress SARS-CoV-2 test analyzed on the GeneXpert (N=265) and Infinity (N=5954) platforms (Cepheid, Sunnyvale, CA), the ID Now (N=53), (Abbott, Chicago, IL) and the Panther Fusion (N=26) (Hologic Inc., San Diego, CA) instruments. Commercial molecular viral assays were validated and performed at NYP Hospital Laboratories following manufacturers’ recommendations.
Patients
The NYP laboratory SARS-CoV-2 testing dataset contains basic demographic information on each patient, including age, gender, race, and health care encounter type and location at the time of order. To better understand SARS-CoV2-2 test utilization and investigate the association with clinical information to the SARS-CoV-2 test results on a subset of patients, we merged our laboratory dataset with the Columbia COVID-19 CARE clinical database.
The COVID-CARE clinical database is an interdisciplinary database, managed by the Columbia Division of Infectious Diseases and contains data on all patients tested for SARS-CoV-2 at the New York Presbyterian West Campus (Milstein Hospital, Allen Hospital, Morgan Stanley Children’s Hospital). This interdisciplinary database contains the work of numerous divisions within the Departments of Medicine, Pediatrics, Neurology, and Obstetrics and Gynecology. The database is comprised of a general primary instrument and multiple sub-specialty instruments resulting in over 720 curated fields per patient. Data was extracted from the medical record through manual review by a team of medical students and sub-specialty fellows and attendings. Manually curated data is combined with structured data from the electronic medical records, pharmacy, and laboratory systems to complete the dataset. As of May 1st 2020, 1,624 patients had undergone manual review (Supplementary Table 1).
Clinical factors and other patient characteristic are compared between SARS-CoV-2 positive and SARS-CoV-2 negative patients in Supplementary Table 2, and between repeat-tested and single-tested patients in Supplementary Table 3. This study was approved by the Institutional Review Boards of Columbia and Cornell Universities.
Data Analysis
Categorical data was presented as frequency and percent. Continuous data was presented as mean (standard deviation) or median and range. Differences between groups were compared using Chi-square tests for categorical variables, and linear ANOVA for continuous variables. Correction for multiple testing was performed using the false discovery rate method of Benjamini-Hochberg (6). Confidence intervals for sensitivity and negative predictive values were calculated using the epiR R package. Time to event using Kaplan Meier curves were used to investigate time to conversion from initially negative to positive and from initially positive to negative. All statistical analyses were performed using the R statistical language (7).
Results
A total of 18,906 assays were performed once per patient and 8,471 assays were performed in 3,432 (15.4%) patients with more than one test completed in the span of 1 to 49 (median = 8) days between assays. There were 2,280 patients with multiple SARS-CoV-2 molecular assays performed who had a first negative, invalid, or indeterminate result and 797 patients who had an initial positive result.
Demographic and clinical characteristics
Patients with any positive SARS-CoV-2 result differed from those that were never positive for SARS-CoV-2 even with repeat testing (Supplementary Table 2). Notably, SARS-CoV-2 positive patients were more likely to be male, older than 44 years, and of self-reported African-American or Hispanic / Latino ethnicity and less likely to be Asian or Caucasian (Supplementary Table 2, all p<0.001). Not surprisingly, severe disease was more likely in SARS-CoV-2 positive patients, as indicated by a higher fatality rate (6.2% vs. 3.8%, p=0.048), presence of symptoms (91.8% vs. 72.4%, p=0.004), need for intubation (10.7% vs. 7%, p=0.026) and frequency of decompensation, as defined by an outcome of intubation, death, or discharge to hospice (13.4% vs. 8.4%, p=0.005).
To determine which factors were associated with repeat testing for SARS-CoV-2 we analyzed demographic and clinical features of repeat-tested as compared to single-tested patients (Supplementary Table 3). Our data show higher age (median = 59.9 vs. 53.4, p<0.001), higher frequency of male gender (52.2% vs. 44.3%, p<0.001), and different distribution of self-reported race and ethnicity in repeat-tested as compared to single-tested patients, with African-Americans and Hispanics / Latinos being more likely to be repeat-tested (p<0.001). At the time of the first test order, admitted patients were significantly more represented in the repeat-tested population in contrast to patients visiting the emergency department and outpatients (p<0.001). Compared with single-tested patients, repeat-tested patients were 3.2 times more likely to be admitted to the ICU (29.7% vs. 9.4%, p<0.001), 4 times more likely to be intubated (24.4% vs. 6.1%, p<0.001), 3.4 times more likely to decompensate (27.6% vs. 8.1%, p<0.001) and 1.7 times more likely to die during the observation period (8% vs. 4.7%, p=0.038).
SARS-CoV-2 Test Performance
The characteristics of the SARS-CoV-2 tests performed in repeat-tested compared to single-tested patients are described in Supplementary Table 4. The vast majority of tests were performed with the cobas 6800 and the use of the various assays was not significantly different between repeat and single-tested patients, except for a small number of patients initially tested with the Thermo-Fisher 7500 assay, which was more likely in repeat-tested patients (p<0.001, standardized Pearson residuals for repeat-tested patients = 3.9).
Among all the repeat-tested patients, 23.2% were positive on the first test (26.5% when indeterminate results were included). If a negative test was repeated on the first day, the positivity rate increased to 26.4% (29.7% with indeterminate results). Overall positivity among repeat-tested patients over the course of the study period was 39.9% in contrast to 49% for single-tested patients (p<0.001). When indeterminate results were counted as positive, 42.9% of repeat-tested patients were positive over the course of the study period in contrast to 50% of single-tested patients (p<0.001).
Indeterminate results are generally considered presumptive positive and occur when only one of two molecular targets is detected. In our repeat-tested patients with an initial result of “Indeterminate”, 54.4% ultimately had a result of “Detected”, as compared to 7% that remained indeterminate upon repeat testing and 38.6% that converted to a negative status during our study period, thus suggesting that it is acceptable to consider these patients positive. However, it is unclear whether the initially indeterminate patients that converted to negative were false positives or presented with low viral loads.
In contrast, a result of “Invalid” reflects the failure to amplify the built-in control and is likely related to poor sampling or inadequate RNA extraction usually due to high viscosity of the samples. In our study of repeat test patients, 52.7% ultimately became positive, 1.2% repeated as “Indeterminate” and 45.6% repeated as ‘Not Detected’. For the analysis of clinical sensitivity of SARS-CoV-2 molecular tests, we considered patients with invalid results as “negative”, as these results can be considered clinically false negatives in the sense that the patient may be infected and the test failed to yield a positive result. In practice, invalid results should always be repeated, preferably with a new sample, as the results are unpredictable.
Initial negative, invalid, or indeterminate SARS-CoV-2 test results were much more frequent among repeat-tested patients (Supplementary Table 4, p<0.001) and conversely, patients without a positive initial result were more likely to be repeated (21%) than initially positive patients (8%, p<0.001).
A subset of the cobas 6800 tests (N=5,343) had cycle threshold (Ct) values available for analysis. The Ct represents the PCR cycle, interpolated to two decimal digits, at which the real-time fluorescent signal crosses a pre-defined threshold for positivity. The Ct is inversely proportional to the concentration of viral RNA. The cobas SARS-CoV-2 RT-PCR assay amplifies two specific targets: Target 1 is located in the ORF1ab non-structural region that is unique to SARS-CoV-2 and Target 2 is a conserved region of the structural protein envelope E-gene common to all members of the Sarbecovirus sub-genus of coronavirus, which include SARS-CoV-2 and SARS-CoV (8, 9). The cobas assay also includes an internal control for assay performance that has no homology to the coronaviruses.
When comparing positive and indeterminate results from repeat-tested with single tested patients, there were no differences in indeterminate Ct values, which is expected as by definition the indeterminate results represent high Ct values. In contrast, the repeat-tested group (N=795) had significantly higher target 2 Ct values (median = 29.1 vs. 27.3, p<0.001 and frequency of target 2 Ct’s above 30 (45.8% vs. 36.9%, p<0.001) compared to single-tested patients (N=4,548), indicating lower viral load in the repeat-tested samples (Figure 1 and Supplementary Table 4).
Analysis of conversion rates of repeat-tested patients
In this study, we classified repeat-tested patients in two groups according to their initial SARS-CoV-2 results: those that were initially positive, for whom repeat testing was most likely intended to ascertain recovery and non-infectiousness, and those that had an initial result of negative, indeterminate, or invalid, in whom persistent clinical suspicion for COVID-19 likely motivated repeat ordering of the test. Supplementary Table 4 shows the different clinical characteristics between repeat-tested and single-tested patients.
For the time-dependent analysis of conversion rates, we considered “initially positive” patients with any “Detected” or “Indeterminate” SARS-CoV-2 result obtained during the first calendar day of testing rather than the first positive test, to reduce bias due to nasopharyngeal sampling inadequacy (Table 1). Conversely, patients without a result of “Detected” or ‘Indeterminate” in the first day were labeled as “initially negative”. Among the 2,413 initially negative repeat-tested patients, 18.6% became positive upon repeat testing on subsequent days (Supplementary Table 5), indicating a negative predictive value of 81.3% (95% CI = 79.7 to 82.8) in this repeat-tested population with the prevalence present at the time of the study.
In a separate analysis, we compared the results of the first test with the results from tests repeated the same day (Supplementary Table 5) or repeated any time after the first result (Supplementary Table 6). Among the patients with repeat testing who had initial results of “Invalid” (241), “Not Detected” (2280), or “Indeterminate” (114), 4.2% had ‘Detected’ results upon repeat testing in the first day of testing (Supplementary Table 5). This increase in positivity rate most likely results from a false-negative initial test due to pre-analytic factors such as sample inadequacy, incorrect swabbing technique, or stochastic sampling bias from low viral loads in the patient nasopharynx. After the first day of testing, repeating invalid, negative, or indeterminate results on the same day resulted in about 0.8% of additional positive results per day, for a total of 18.4% positives that were missed by the first test.
Among the 1,371 repeat-tested patients with one or more SARS-CoV-2 results of “Detected”, which can be assumed to be truly infected, only 58.1% were resulted as “Detected” on the initial test (Supplementary Table 5), and only 66.2% were reported as “Detected” on the first day (Table 1). Considering ‘Detected’ and “Indeterminate” as positive, 1,471 repeat-tested patients had one or more SARS-CoV-2 positive results over time; only 61.9% were positive on the initial test and only 69.3% had a positive result on the first day (Table 1). These data provide an estimate of the clinical sensitivity of the assay in the repeat-tested population, and establish a baseline to look at conversion rates from negative to positive.
Table 2A shows the number of patients who had an initial result of “Not Detected” on day 1 who converted to a SARS-CoV-2 positive status grouped per time after the initial test. Conversely, Table 2B shows rates of conversion to negative for patients with a status of “Detected” on day 1. Supplementary Figure 1 shows the distribution per day after onset of symptoms (a and b) or after initial testing (c to f) of conversions from positive to negative (left-side) and from negative to positive (right-side) in the two groups of repeat-tested patients. For this analysis, positive status included “Detected” and “Indeterminate” and negative status included results of “Not Detected” with “Invalid” results excluded. Among the initially positive patients, the unadjusted distributions show a peak of conversion to negative between 30 and 40 days after symptoms or around 20 days after initial testing. Less than 10% of the patients who converted to negative converted before 15 days after onset of symptoms or 10 days after initial testing. In contrast, among the patients with initially negative results who converted to positive, most conversions occurred 10 days or less after onset of symptoms, and in the first 1-3 days after initial testing.
Since we cannot be certain about the conversion rates due to a significant proportion of repeat-test patients having insufficient testing performed to detect conversion (right-censoring) we used a Kaplan-Meier approach to estimate the conversion rate by day of testing (Figures 2, 3 and Supplementary Figure 1 e to h) with the following assumptions:
When there were multiple tests performed per patient in one day, the results were aggregated to the highest result, i.e. Detected > Indeterminate > Not Detected.
For initially positive patients, an event is defined as highest result in a day of “Not Detected”.
Conversely, an event is defined as a conversion from a “Not Detected”, or “Invalid” result at day 1 to “Detected” or “Indeterminate” on subsequent days.
If the last test result was unchanged relative to the first day result, the patient was considered right-censored at that time.
Only the first day and either the censoring day or the event day were used for each patient and indeterminate results were ignored.
The results show that the probability of converting from positive to negative in the initially positive, repeat-tested population is minimal until about 15 to 20 days after initial testing and reaches 50% at 28 days (95% CI = 27 to 29 days, Figure 2). In the initially negative repeat-tested population, the risk linearly increases every day with a 25% probability of conversion to positive of 20 days (95% CI = 16 to 23 days, Figure 3).
Since lack of conversion could be due to death of the patient, which occurred in about 8% of the repeat-tested population, we performed a competing risk analysis with death as the alternative event, using the timereg R package (10), and did not find significant differences in either the positive to negative or the negative to positive Kaplan-Meier cumulative probability of conversion (results not shown).
Interestingly, among the repeat-tested patients we have identified 11 patients that converted from an initial SARS-CoV-2 negative result to positive and back to negative over several days. We also identified 49 initially positive patients that became negative and later turned positive again. The time course of SARS-CoV-2 test results for these patients are shown on Supplementary Figure 2.
Discussion
Our results from repeat-tested patients can be used to estimate the clinical sensitivity of the SARS-CoV-2 molecular testing in the population of patients that were selected for repeat testing, in contrast to the general population of tested patients. With all likelihood, most repeat testing on initial negative patients was performed either to follow a history of exposure, when the clinical profile did not fit the initial results, or when clinical presentation deteriorated after the initial result. Consistent with this hypothesis, repeat-tested patients were more likely to be older, male and of non-Caucasian race than single tested patients (Supplementary Table 3), consistent with the demographics of COVID-19 positive patients (Supplementary Table 2). Importantly, repeat-tested patients had worse outcomes as demonstrated by higher rates of decompensation, intubation, and mortality. Interestingly, repeat-tested patient who converted from positive to negative tended to be younger and present as outpatients, as compared to patients that remained positive, but there were no significant differences in clinical outcomes (Supplementary Table 7). In contrast, repeat-tested patients that remained negative tended to be inpatients, have longer admission duration, and were more likely to be extubated than those who converted to positive (Supplementary Table 8), suggesting that a significant number of repeat-testing in SARS-CoV-2 negative patients was performed in inpatients admitted before the pandemic or for non-COVID-19 reasons. Indeed, repeat-tested patients admitted before March 1st, 2020 were much more likely to have a persistent negative result (71.4%) than those admitted after March 1st, 2020 (39.7%, p<0.001).
In the absence of a more sensitive gold-standard, the repeat-tested patients with an eventual positive result can be considered true positives, as the analytical specificity of molecular testing is very high (3, 8, 9, 11, 12). It may be tempting to add all the 9,272 single-tested positive patients to the 1,371 repeat-tested positive patients to determine clinical sensitivity. However, we don’t know how many of the “Not detected” single-tested patients are false negatives, especially given the high frequency of asymptomatic or mildly symptomatic COVID-19 patients (13-15). Therefore, considering only the positive patients will inflate the estimated clinical sensitivity. Nevertheless, if we consider all negative results (repeated or not) to be true negative (i.e. a specificity of 100%) we can estimate the upper bound of the clinical sensitivity of a first initial result to be 94.6% (95% CI = 94.2-95%). If the test is repeated on the first day to account for nasopharyngeal sampling inadequacy, the upper boundary of the estimated sensitivity with these assumptions would be 95.6% (95% CI=95.2-96%). A lower bound of clinical sensitivity can be estimated by considering the worst case scenario that the same percentage of false negatives identified in repeat-tested patients would apply to the general population. With these assumptions, the clinical sensitivity of the SARS-CoV-2 assay can be estimated to be between 66.2% (95% CI = 63.6 to 68.7) and the above best case estimate of 95.6%.
The lower clinical sensitivity of the first day results in the ultimately positive repeat-tested patients suggests several possibilities:
Viral shedding increases over time in a recently infected patient and will eventually cross the detection threshold in subsequent samples. This possibility is suggested by the lower Ct values for target 1 and particularly target 2 on the repeat-tested patients (Supplementary Table 4 and Figure 1).
A significant number of samples are improperly collected, and repeat testing increases the probability of detection; this is particularly likely in the first day of testing, when suspicion may be high but the initial test results were negative or inconclusive, as shown in Supplementary Table 5.
Initially tested patients were truly negative and acquired the infection nosocomially after admission.
The mean interval between an initial negative test result and the first positive in patients who converted in our study was 9.4 days (95% CI = 8.4 to 10.5 days, N = 335), which is longer than reported by Ai et al, who showed a mean interval time between initial negative to positive RT-PCR results of 5.1 ± 1.5 days with a median of 4 days (N = 15) (4). We have insufficient data to calculate incubation time as only 1.3% of the patients had symptoms after the first test, with a median time between symptoms and testing of 1.3 days. The median interval between start of symptoms and the first test in our study of 4.8 days (95% CI = 4.5 to 5.4 days) is in line with the data from Lauer et al., who calculated a median incubation time of 5.1 days (95% CI = 4.5 to 5.8 days) in patients from China (16).
The third possibility of hospital-acquired infection explaining a low rate of initial true negative result is less likely because a) patients who converted from negative to positive had shorter intervals between admission and the first test and shorter duration of admission than those who remained negative (Supplementary Table 9 and b) there were no significant differences between repeat-tested inpatients and outpatients in the average time interval between the initial negative test and the first positive test (results not shown). If there were a large number of inpatients acquiring the infection in the hospital one would expect longer intervals between admission and positivity due to the cohort of patients already admitted before the pandemic, as compared to patients recently admitted with COVID-19. Rather, these data suggest a pattern of repeated ordering in uninfected inpatients with a lower likelihood of conversion.
Our analysis of repeat-tested patients with an initial positive result, using the Kaplan-Meier estimator, indicates that conversion to a negative result is unlikely to occur until about 15 to 20 days after initial testing or 20 to 30 days after start of symptoms, when the odds ratio significantly increase (Figure 2 and Supplementary Figure 1). Conversely, when a patient is suspected of COVID-19 but the initial test is negative, repeating the test steadily increases the probably of conversion to ‘Detected’ every day (Figure 3). Repeat testing is especially likely to yield a positive result if the initial test is indeterminate or invalid. For indeterminate results, this likely reflects low levels of the virus in the sample or low viral shedding in the nasopharynx early in the course of the infection. For invalid results, this likely reflects sampling inadequacy, often due to excess mucous in the sample. Our finding of significantly higher Ct in repeat-tested patients as compared to single-tested patients supports this hypothesis.
Limitations of the study
This is an observational study without selection bias for the laboratory data, as all results were included in the analysis. However, the clinical data was restricted to a subset of patients seen at Columbia University Irving Medical Center campuses. Nevertheless, rates of positivity and demographic variables captured in the laboratory dataset were not significantly different between the other campuses, suggesting that the population in the clinical dataset is generally representative of the New York City patients tested for COVID-19. Other limitations of the study of repeat-tested patients include: 1) only 15% of the total patient’s tested had repeat testing done and 2) ordering of repeat testing was at the discretion of the health care providers and not performed according to a standard protocol, although test ordering was mostly accomplished through standardized electronic medical record order sets.
Conclusions
Our data suggest that patients with a high clinical suspicion or exposure setting suggestive of COVID-19 should be tested by a molecular SARS-CoV-2 assay and it is appropriate to repeat the test the same day or in subsequent days if the results are initially negative. Conversely, for patients with a positive SARS-CoV-2 molecular assay result, repeating the test before at least 15 days after the first test is unlikely to yield a negative result. Whether repeat positivity represents active infection or detection of nonviable viral RNA is unknown. Further studies are needed to develop predictive models of the course and outcomes of COVID-19 using well-curated demographic, clinical, and laboratory datasets.
Data Availability
The anonymized data stripped of personal health identifiers sufficient for reproducing the results in this manuscript will be shared upon request and review by NYP, Cornell and Columbia Universities.
Funding
This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.
Potential conflicts of interest
All authors have no conflicts.
Supplementary Tables
Acknowledgments
We would like to acknowledge the contributions of all the dedicated medical technologists and laboratory technicians who performed testing at NYP laboratories, the efforts of the NYP Laboratory Information Team, in particular Kelvin Espinal, Sarah Russell, Dennis Camp, Yingzhe Kuang, and Bulent Oral for designing queries and providing data extracts, and the volunteers who helped gather data from electronic medical records for the COVID-19 CARE database.