Multicentre evaluation of two multiplex PCR platforms for the rapid microbiological investigation of nosocomial pneumonia in UK ICUs: the INHALE WP1 study

Patients and specimens

Between September 2016 and May 2018, surplus routine lower respiratory tract samples were collected from eligible patients with suspected HAP/VAP across the 15 ICUs participating in the INHALE study. These sites were selected in order to represent a range of UK institution types, and included tertiary referral centres, district general hospitals, one children’s hospital and one private hospital. They comprised Aintree University Hospital NHS Foundation Trust, Bupa Cromwell Hospital, Chelsea and Westminster Hospital NHS Foundation Trust, City Hospitals Sunderland, Dudley Group NHS Foundation Trust, Great Ormond Street Hospital, Guy’s and St Thomas’ NHS Foundation Trust, James Paget University Hospitals NHS Foundation Trust, Norfolk and Norwich University Hospitals NHS Foundation Trust, North Middlesex University Hospital NHS Trust, Queen Elizabeth Hospital Kings Lynn NHS Trust, Royal Free Hospital, Royal Liverpool and Broadgreen University Hospitals NHS Trust, University College London Hospitals and University Hospitals of North Midlands and they were served by 11 different microbiology laboratories

Specimens were included if they were of sufficient volume (>400 μl) and were from ICU inpatients hospitalized for at least 48h who were about to receive a new antibiotic or change in antimicrobial therapy for suspected lower respiratory tract infection. Specimens were eligible for inclusion only when collected from the patient within 12h (before or after) of antimicrobial therapy being administered and then tested, or frozen at −80°C, within 72h of collection. All types of lower respiratory specimens were accepted, but upper respiratory tract specimens (e.g. nasopharyngeal aspirates) were excluded. Second specimens from the same patient were included only when collected > 14 days after the first sample.

Ethical approval

This work had study specific approval from the UK Health Research Authority (Reference: 16/HRA/3882, IRAS ID: 201977,) and the UCL DNA Infection Bank Committee whose operation is governed by the London Fulham Research Ethics committee (REC Reference: 17/LO/1530).

Conventional culture and susceptibility testing

Each respiratory specimen was initially cultured locally, at the laboratory serving the participating hospital, according to their standard operating procedures (SOPs). These SOPs were all based on the Public Health England (PHE) UK Standard.¹⁷ Except in the case of bronchoalveolar lavage (BAL) specimens this specifies initial homogenisation of the respiratory sample with 0.1% dithiothreitol, followed by a 10⁻⁵ dilution, and inoculation of the diluted and undiluted specimen onto chocolate agar with bacitracin (incorporated or as a disc), cysteine lactose electrolyte deficient agar (CLED) or MacConkey agar, and Sabouraud agar for fungi. Blood agar may also be added, especially if bacitracin is incorporated in the chocolate agar. In the case of Bronchoalveolar lavage (BAL), culture is performed on serial dilutions of a sample that has been concentrated by centrifugation.

Plates are incubated at 35-37°C in the presence of 5% CO₂ (blood and chocolate agar) or in air (MacConkey and CLED agar) for 40-48h, with daily reading of results. Bacterial pathogens are identified to species level by MALDI-TOF or biochemical methods, followed by antimicrobial susceptibility testing using EUCAST or BSAC interpretive standards.

The PHE standards provide guidance on the interpretation of culture results for BAL samples, but interpretation and reporting are left to the discretion of individual laboratories for other sample types.

PCR Testing

Samples were transported to the two central research laboratories (University of East Anglia and University College London, Royal Free Hospital campus) by courier. Upon receipt, each specimen was promptly tested using both the Unyvero Pneumonia Panel (Curetis, Holzgerlingen, Germany) and the BioFire FilmArray Pneumonia Panel (BioFire Diagnostics, Salt Lake City, USA) according to manufacturer’s instructions. The basic characteristics of each test, along with the panel of targets sought are detailed in Table 1.

Supplementary analyses

All specimens with a sufficient surplus (300 µl) after PCR testing underwent 16S metagenomic analysis. Samples were inactivated by incubating for 30 minutes at 99°C, then DNA was extracted using the ZR Viral RNA/DNA kit and ZR BashingBead Lysis Tubes (Zymo Research). Briefly, 300 µl of sample were transferred in a bead tube, homogenized in a bead-beater for 30 seconds at 3,500 oscillations per minute, centrifuged for 1 minute at 21,000 g. Next, 200 µl of supernatant were transferred to a clean Eppendorf tube and DNA was extracted following manufacturer’s instructions. Illumina 16S metagenomic sequencing was then performed according to the manufacturer’s protocol (Illumina, 15044223B): the V3-V4 16S rRNA region was amplified on a LightCycler 480 II instrument (Roche) and sequenced on an Illumina MiSeq system. The Illumina BaseSpace 16S metagenomic pipeline was used to analyse the results. Only samples with at least 10,000 total reads were deemed eligible for analysis. For a genus to be considered significant, it had to comprise at least 1% of all reads.

A sub-set of specimens, selected at random among those where culture and PCR disagreed for resistance detection underwent additional culture-based analysis, termed comprehensive culture (CC) at the UCL research laboratory, using methodology described previously.¹⁵ Briefly, a sweep of growth was taken across the plate of a fresh primary culture of the specimen on chocolate agar and stored in Microbank^TM vials at −80°C until analysis. Ten microliters of neat sample and a 10⁻⁵ dilution in 0.9% saline were then plated onto chocolate agar, Columbia blood agar (CBA), Brilliance UTI agar (Oxoid, Basingstoke, UK) and Columbia colistin-nalidixic acid agar (C-CNA) (Oxoid). CBA, UTI and C-CNA plates were incubated at 37°C in air for 18h; chocolate agar plates were incubated in 5% CO₂ at 37°C for 18h. Representative bacterial colonies of different morphologies on each medium were identified by MALDI-TOF MS (Bruker GmbH, Mannheim, Germany), either directly from colonies or by using formic acid extraction where necessary.

Identification of antimicrobial resistances

Additional investigation of antimicrobial resistances, or the genes responsible was performed on isolates found resistant in microbiology laboratories or CC or when either of the two molecular systems detected key resistance genes.

Gram-negative bacteria (i) reported resistant to cephalosporins or carbapenems in RM, (ii) found to have ESBL or carbapenemase genes using the PCR systems, or (iii) grown in CC were tested for resistance to ceftazidime, cefotaxime, ceftriaxone, ertapenem, meropenem and imipenem (Enterobacterales) or imipenem, meropenem, ceftazidime and piperacillin/tazobactam (Acinetobacter spp. and P. aeruginosa) by EUCAST disc diffusion methodology.¹⁸ Potential methicillin-resistant Staphylococcus aureus (MRSA) were screened for resistance to cefoxitin.

When isolates had phenotypes consistent with the presence of antimicrobial resistance genes, further genetic testing was performed. Enterobacterales resistant to a carbapenem or to oxyimino cephalosporins, P. aeruginosa resistant to both carbapenems and cephalosporins and A. baumannii resistant to imipenem or meropenem were tested with the Check-MDR CTX103XL kit (Checkpoints, Wageningen, the Netherlands) according to manufacturer’s instructions, following extraction of total genomic DNA using the Qiagen DNA Mini Kit (Qiagen). S. aureus isolates resistant to cefoxitin underwent in-house PCR (primers and conditions described previously)^19,20 for detection of mecA and mecC using HotStartTaq PCR Mastermix (Qiagen) on DNA extracted with the Qiagen DNA Mini Kit.

Data collection

RM data available on the Laboratory Information Management Systems (LIMS) of each participating hospital was were collected and managed using REDCap²¹ electronic data capture tools hosted at Norwich Clinical Trials Unit. For each included sample, we collected the culture result as reported to treating clinicians and details of significant organisms reported and their full antimicrobial susceptibility profile. Any results for relevant respiratory pathogens detected by non-culture-based methods were also included. Virology data (PCR) were collected if testing had been performed on the same calendar day as collection of the lower respiratory tract sample. We also collected details required to confirm patient eligibility and times samples were collected, processed and results released. All PCR and supplementary data generated by study staff were also recorded in RedCap. All data were anonymised.

Data Analysis

Analyses were carried out using Stata (v 15) and R (v 3.5 or above) and followed a pre-defined, detailed statistical analysis plan. Results from the conventional and PCR tests, including about timings, detection of individual pathogens, viruses and fungi were described using standard summary statistics. We considered agreement between results from RM and the PCR tests by categorising each sample in terms of overall concordance of organisms detected. Definitions of the categories are detailed in table 2. For each PCR test, the proportions in the concordance categories were calculated, with 95% confidence intervals.

For both PCR tests, sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) were estimated (with exact 95% confidence intervals) for each target using RM and routine virology as the gold standard. Owing to concerns that RM provides a poor gold standard (see Results), estimates (with 95% credible intervals) were also calculated using Bayesian latent class (BLC) models²² incorporating results from both PCR tests, 16S metagenomics and RM. Models used non-informative priors for all parameters (except to constrain specificities above 0.15 to obtain more stable posterior distributions) and were fitted with and without assuming correlation between tests. The best fitting models were identified based on Deviance Information Criteria (DIC).

Scoring system for evaluation of PCR-based Diagnostic Tests

At the outset of the study, through expert consensus, a scoring system was developed in order to assess the suitability of each ‘sample-in, answer-out’ test for use in the INHALE randomized controlled trial and more generally, for routine diagnostic use. Tests were assessed against one essential criterion which had to be met for further evaluation, and 10 points-based ‘Desirable Criteria’, scoring a total of 150. The essential criterion stipulated that incidence of major discordances, defined as failures to detect pathogens found by RM, was less than 5%.

Desirable Criteria were: (i) overall concordance (max. 45 points), (ii) sensitivity (20 points), (iii) failure rate (15 points), (iv) breadth of panel (15 points), (v) time to result (15 points), (vi) cost per test (15 points), (vii) footprint (5 points), (viii) consumable logistics (5 points), (ix) quality of customer service (5 points) and (x) ease of use (10 points). Criteria i-iii were scored based on study results, criteria iv-viii on manufacturer’s published information and criteria ix and x on a user questionnaire. The scale was deliberately weighted towards characteristics related to detection of pathogens, with implementation-based criteria given a lower weighting. Full details of the scoring system can be found in supplementary data (Table S1).

Specimens Collected

A total of 752 specimens, 652 of them eligible, were collected from 15 participating ICUs (Figure 1). The range of eligible samples per site was 7-141, with 9 sites providing >20 eligible samples. Most were from adults, with just 72 (11.0%) from children. Among all samples, 260 (39.9%) came from patients with suspected HAP and 392 (60.1%) from patients with suspected VAP. Endotracheal aspirates were the most common sample type, accounting for 299 samples (45.9%); followed by sputa (272 samples, 41.7%), BALs (44 samples, 6.7%) and non-directed BALs (23 samples, 3.5.%), with the remaining 14 samples in the “other” or “unknown” category. A small majority of samples were collected before antibiotic administration (357, 54.8%).

• Download figure
• Open in new tab

Figure 1.

Schematic representation of sample eligibility

Routine Microbiology Results

RM culture was performed at the local laboratories on all samples. The overall median time to a result was 70.2h (interquartile range (IQR) 51.1h to 92.1h), including a median of 6.1h (IQR 2.5h to 15.4h) transit time from the ICU to booking in at the laboratory and 55.5h (IQR 44.8h to 76.5h) from sample booking to release of results. The positivity rate was 44.2% and most positive results recorded just one significant organism (35.1%), with a minority reported as containing two or more (9.1%). The remaining 55.8% were reported mostly as ‘normal flora’, ‘non-significant growth’, or simply as no ‘growth’.

S. aureus was the most-frequently-found bacterium (Figure 2), representing 23.6% (83/352) of all organisms reported, closely followed by P. aeruginosa at 20.7 %; Enterobacterales collectively accounted for 38.1% of isolates with Klebsiella spp. (12.5% of all isolates) and E. coli (12.2%) both prominent (Figure 3a). Although H. influenzae is not primarily considered a hospital-acquired organism, it accounted for 6.5% of organisms cultivated. Occasionally RM laboratories reported Candida spp., Enterococcus spp. and coagulase-negative staphylococci. Because there is no evidence base for their involvement in pneumonia, they were not included in the analyses.

• Download figure
• Open in new tab

Figure 2.

Number of significant organisms detected per respiratory sample by RM or PCR.

• Download figure
• Open in new tab

Figure 3a.

Numbers and types of bacteria detected by RM culture from respiratory samples included in the study.

Results of standard-of care for respiratory virus diagnostics were recorded if testing was performed in the same 24h period as collection of the eligible bacteriology specimen. Only 113 patients, 33 of them children, had a virology result meeting this eligibility criterion, and, of these, 31 (27.4%) were positive. The most-frequently-identified viruses were influenza A (n = 7), adenovirus (n = 6) and cytomegalovirus (n = 6).

PCR Results

Of 652 eligible samples, 631 tested on the Unyvero system were eligible for further analysis and 632 of those tested on the FilmArray were eligible (Figure 1). The remainder were excluded owing to not being tested or frozen within 72h of collection. Among the eligible samples, 620 generated a result on the FilmArray, whilst tests on the remaining 12 failed, a test failure rate of 1.9%. Defining failure on the Unyvero is more complex since target organisms and genes are divided into 8 chambers and either the entire test or one or more chambers may fail. In the latter case, useful results may still be generated from the remaining chambers. We considered any test where >2 chambers failed as a “total failure”. In a further 32 cases just 1 or 2 chambers failed and data from the remaining 6 or 7 chambers were included in the analysis for these partial failures, with the proviso that organisms sought by the failed chambers may have been missed. A further 24 samples generated other types of total machine failure (unrelated to the chambers) which failed to generate a result for the Unyvero; altogether, there were 606 valid results, a total failure rate of 4.0%.

The overall positivity rate of both PCR tests was higher than RM, at 60.4% for the Unyvero and 74.2% for the FilmArray (chi-square test: p < 0.0001). Most specimens had multiple organisms detected (Figure 2), with this proportion higher for FilmArray than the Unyvero. FilmArray found only bacteria in 54.2% of samples and only viruses in 6.9% whereas 13.1% contained both viruses and bacteria. The most common bacterial pathogens detected by the two PCR tests are shown in Figure 3b. The principal species and their relative prevalence were broadly similar to that recorded by RM, although E. coli and Klebsiella spp. were detected more frequently in relative terms by PCR than by culture, whereas S. aureus and P. aeruginosa were detected less frequently. Between them, these four species were the most-commonly-detected organisms by both PCR and RM. Among viruses detected by the FilmArray (Unyvero doesn’t seek viruses), rhinovirus was the most prominent (n=55), followed by influenza A (n=29) and influenza B (n=25) (full viral detection data are shown is Table S2).

• Download figure
• Open in new tab

Figure 3b.

Numbers and type of bacteria detected by PCR from respiratory samples included in the study. Unyvero, solid bars, n = 606; FilmArray, hatched bars, n = 620. Species sought by one test only are marked with an asterisk.

Performance of PCR Tests

The performance of the PCR tests was measured and compared in several ways, in order to accommodate for the fact that RM is an imperfect gold standard. Moreover, the PCR tests detect multiple targets, more than one of which may be present in any given sample. Consequently, it is challenging to calculate overall test sensitivity and specificity; it was commonplace, for example, for PCR to detect two organisms, one of which matches an organism reported by RM and the other does not.

First, overall test performance was measured in terms of concordance with RM, as shown in Table 2. Both PCR tests deliver semi-quantitative outputs; the FilmArray reports bacterial targets as 10⁴, 10⁵, 10⁶ or ≥10⁷ copies per ml, whereas the Unyvero reports pathogen targets as +, ++ or +++. We therefore further calculated concordance taking into account only those targets that were detected at high concentration, defined as 10⁶ or ≥10⁷ copies/ml for FilmArray and ++ or +++ for Unyvero. These results are also included in Table 2.

Around one half of PCR results by either method demonstrated full positive or negative concordance with RM when all PCR detections were considered. Most of the remainder were either partially concordant or had minor discordance, predicated on PCR reporting more organisms than RM. Major discordance was rare, totalling only 4.6% of Unyvero results and 1.8% of FilmArray results. If PCR detections at low concentrations were excluded, full concordance increased for both tests but major discordance also increased substantially. Target-specific sensitivity data are shown in Table 3. Local microbiology laboratories do not normally seek atypical pathogens in HAP/VAP patients, so results for Chlamydophila pneumoniae, Legionella pneumophila and Mycoplasma pneumoniae are excluded from sensitivity analyses. Unyvero and FilmArray each detected M. pneumoniae once, in the same specimen, where it was not sought by local microbiology. Unyvero detected two samples with L. pneumoniae; FilmArray and RM did not detect any. None of the three methods detected C. pneumophila.

Owing to the small number of routine virology results available for comparison, confidence intervals for viral detections were wide (Table S2, FilmArray only, as Unyvero does not seek viruses).

For most target bacteria, PCR assay sensitivity was >95% and NPV was > 98%. Specificity and PPV were lower, due to the PCR tests detecting more organisms per sample and finding more positive samples than RM. Notably, both machines often found the same organism as each other when RM did not record any organism. Such agreement is unlikely to be random and suggests RM false negative rather than PCR false positive results may represent an imperfect gold standard. Accordingly, BLC modelling was introduced, rather than gold standard comparison, to deliver a more representative comparison of all tests. Table 4 shows performance estimates obtained from BLC models that do not rely on RM as a gold standard. Based on DIC values, models that assumed independence of the test results were preferred.

Based on this BLC analysis, RM was the least sensitive technique for pathogen detection, with sensitivity values for individual pathogens ranging from 27.1 % for Streptococcus pneumoniae to 68.7% for P. aeruginosa. In contrast, sensitivity values for the PCR tests remained high; FilmArray sensitivity ranged from 89.4% to 99.3 %, whereas Unyvero sensitivity values ranged from 83.9% to 96.9% (expect K. aerogenes, 48.4%). Specificity and PPV values for both PCR tests increased considerably compared to the values calculated using RM as a gold standard: in particular, specificity was above 99% for Unyvero targets and ranged from 93.9% to 99.9% for FilmArray targets. The PPV range was 62.1% to 99.3% for Unyvero and 56.1% to 96.6% for FilmArray. These results are consistent with the observation that FilmArray detects pathogens more frequently than Unyvero.

16S Metagenomic Analysis

16S metagenomic analysis was originally included to act as an independent molecular reference method. Four-way BLC analysis including 16S metagenomic results is shown in Table 5. However, the 16S technique is only able to distinguish organisms to genus level, so PCR and RM data are likewise grouped to genus level. Streptococci are omitted because of the high density of commensal streptococci found in the respiratory tract and the inability of the 16S method to distinguish these from each other and from pathogenic species. For this analysis only, Klebsiella aerogenes was grouped within the genus Enterobacter owing to its relatively recent re-classification. The results show that 16S metagenomics was less sensitive than PCR and so was not an effective control method; nonetheless, it had had greater sensitivity compared with RM. Further optimisation may yield better results.

Antimicrobial Resistance

All RM results for antimicrobial susceptibility testing were recorded and Table 5 shows rates of resistance to antimicrobials commonly used to treat HAP and VAP for the most-frequently-isolated species. P. aeruginosa exhibited the broadest resistance, with ≥ 10% rates for relevant antimicrobials. Other organisms, notably E. coli and S. aureus, had high rates of resistance to particular agents - 21.4% of E. coli were resistant to third-generation cephalosporins, and 18.4% of S. aureus were MRSA, however, all isolates of these species retained full susceptibility to other antimicrobials including meropenem for E. coli and glycopeptides for S. aureus. All species/genera had MDR rates of above 10%, with E. coli the highest at 37.1%.

The PCR tests differ from RM by seeking resistance genes in a whole sample, rather than in particular bacteria. Very few ‘high concern’ resistance genes were found. Among carbapenemase genes the Unyvero found 6 acquired carbapenemase genes (2 bla_NDM, 1 bla_KPC and 3 bla_VIM) in addition to 5 A. baumannii bla_OXA-23 genes, whereas FilmArray found 3 acquired carbapenemases (1 each of bla_KPC, bla_IMP and bla_VIM; it does not seen the genes encoding Acinetobacter OXA enzymes). Only two of these detections, one bla_KPC and one bla_VIM overlapped between the two tests. bla_CTX-M, encoding the predominant family of ESBLs now seen in Enterobacterales was found by Unyvero in 14 specimens and by FilmArray in 32, with all Unyvero detections confirmed by the FilmArray. mecA/C, encoding methicillin resistance, was found in the presence of S. aureus by Unyvero in 25 specimens and by FilmArray in 32, with 18 of these in common between the two tests. Unyvero found 70 additional detections of mecA/mecC in the absence of S. aureus, this type of detection is prevented by design in the FilmArray, which amplifies across the junction between mecA and the S. aureus chromosome. Unyvero also seeks a wide range of resistance determinants, compromising unprotected penicillins, macrolides and sulphonamides; detections of these are summarised in Table S6; they were widely found but are more pertinent to antibiotics used in community pneumonia and were not considered further.

Assessment of the machines’ performance in respect of resistance gene detection is difficult because RM often reported no organisms for samples that were PCR positive for an organism and a resistance gene. In other cases, we were unable to retrieve routine clinical isolates for genetic validation. Where possible, RM isolates were independently investigated for the presence of antimicrobial resistance genes detected by the PCR tests. These isolates were supplemented with those recovered from our “comprehensive culture” methodology on a sub-set of discrepant samples; this aimed to recover all viable target species present, regardless of quantity. Following phenotypic antimicrobial susceptibility testing by disc diffusion, ESBL and carbapenemase genes were sought in Gram-negative bacteria using a Checkpoints microarray assay, whereas mecA and mecC genes were sought in S. aureus isolates by PCR. Concordance was then compared. Results are shown in table 6.

Among putative carbapenemase-producers, detections of bla_KPC (n=1) and OXA-23 (n= 5, Unyvero only) were in complete agreement, with the same genes detected in cultivated isolates. FilmArray missed 1/2 cases where the carbapenemase gene bla_VIM was detected in a cultivated isolated, and Unyvero missed 3/15 cases where the ESBL gene bla_CTX-M was present and 1/14 cases of MRSA. On the other hand, Unyvero detected one bla_VIM and two bla_NDM genes that could not be confirmed in cultivated isolates, FilmArray detected one bla_IMP that was not confirmed. Two bla_CTX-M detections by Unyvero, six bla_CTX-M detections by FilmArray, seven mecA/C-MREJ detections by Unyvero and six mecA/C-MREJ detections by FilmArray could not be confirmed by further culture-based investigation. Nine FilmArray detections of bla_CTX-M and 4 Unyvero, 11 FilmArray detections of mecA/C-MREJ were not investigated due to lack of materials and/or resources.

In total, comprehensive culture detected 12 additional organisms with confirmed resistance genes that were unrecorded by RM, which either had not isolated them or had discounted them.

Overall Comparison of PCR Tests

Both PCR systems met the one essential requirement of having <5% major discordances. On this basis we collated all of the test results for both systems and combined them with performance and implementability data in order to choose which to carry forward to the INHALE RCT. Since, in this RCT, the machine would be located in the ICU rather than the microbiology laboratory and would require operation by medical and nursing staff, we sought to ensure that the chosen test was easy to utilise in this environment. Accordingly, our scoring scheme not only evaluated performance but also ease-of-use, footprint, turnaround time, as well as the overall user experience. This scheme was reviewed and agreed by INHALE’s independent Programme Steering Committee (PSC) as well as its Patient and Public Involvement (PPI) panel. The results are presented in Table 7.

FilmArray scored 105 points, compared with 68 for Unyvero. Overall, Unyvero was more concordant with RM but FilmArray had better sensitivity; Unyvero had a broader target panel but more failed tests. The FilmArray clearly performed better on characteristics relating to implementation, ease-of-use, turnaround time and user experience.

Accordingly, we have preferred the FilmArray Pneumonia Panel for the RCT, now being undertaken across 12 UK ICUs (Trial ID: ISRCTN16483855)