TY - JOUR T1 - Real-world evaluation of AI-driven COVID-19 triage for emergency admissions: External validation & operational assessment of lab-free and high-throughput screening solutions JF - medRxiv DO - 10.1101/2021.08.24.21262376 SP - 2021.08.24.21262376 AU - Andrew A. S. Soltan AU - Jenny Yang AU - Ravi Pattanshetty AU - Alex Novak AU - Yang Yang AU - Omid Rohanian AU - Sally Beer AU - Marina A. Soltan AU - David R. Thickett AU - Rory Fairhead AU - CURIAL Translational Collaborative AU - Tingting Zhu AU - David W. Eyre AU - David A. Clifton Y1 - 2021/01/01 UR - http://medrxiv.org/content/early/2021/11/06/2021.08.24.21262376.abstract N2 - Background Uncertainty in patients’ COVID-19 status contributes to treatment delays, nosocomial transmission, and operational pressures in hospitals. However, typical turnaround times for batch-processed laboratory PCR tests remain 12-24h. Although rapid antigen lateral flow testing (LFD) has been widely adopted in UK emergency care settings, sensitivity is limited. We recently demonstrated that AI-driven triage (CURIAL-1.0) allows high-throughput COVID-19 screening using clinical data routinely available within 1h of arrival to hospital. Here we aimed to determine operational and safety improvements over standard-care, performing external/prospective evaluation across four NHS trusts with updated algorithms optimised for generalisability and speed, and deploying a novel lab-free screening pathway in a UK emergency department.Methods We rationalised predictors in CURIAL-1.0 to optimise separately for generalisability and speed, developing CURIAL-Lab with vital signs and routine laboratory blood predictors (FBC, U&E, LFT, CRP) and CURIAL-Rapide with vital signs and FBC alone. Models were calibrated during training to 90% sensitivity and validated externally for unscheduled admissions to Portsmouth University Hospitals, University Hospitals Birmingham and Bedfordshire Hospitals NHS trusts, and prospectively during the second-wave of the UK COVID-19 epidemic at Oxford University Hospitals (OUH). Predictions were generated using first-performed blood tests and vital signs and compared against confirmatory viral nucleic acid testing. Next, we retrospectively evaluated a novel clinical pathway triaging patients to COVID-19-suspected clinical areas where either model prediction or LFD results were positive, comparing sensitivity and NPV with LFD results alone. Lastly, we deployed CURIAL-Rapide alongside an approved point-of-care FBC analyser (OLO; SightDiagnostics, Israel) to provide lab-free COVID-19 screening in the John Radcliffe Hospital’s Emergency Department (Oxford, UK), as trust-approved service improvement. Our primary improvement outcome was time-to-result availability; secondary outcomes were sensitivity, specificity, PPV, and NPV assessed against a PCR reference standard. We compared CURIAL-Rapide’s performance with clinician triage and LFD results within standard-care.Results 72,223 patients met eligibility criteria across external and prospective validation sites. Model performance was consistent across trusts (CURIAL-Lab: AUROCs range 0.858-0.881; CURIAL-Rapide 0.836-0.854), with highest sensitivity achieved at Portsmouth University Hospitals (CURIAL-Lab:84.1% [95% Wilson’s score CIs 82.5-85.7]; CURIAL-Rapide:83.5% [81.8 - 85.1]) at specificities of 71.3% (95% Wilson’s score CIs: 70.9 - 71.8) and 63.6% (63.1 - 64.1). For 3,207 patients receiving LFD-triage within routine care for OUH admissions between December 23, 2021 and March 6, 2021, a combined clinical pathway increased sensitivity from 56.9% for LFDs alone (95% CI 51.7-62.0) to 88.2% with CURIAL-Rapide (84.4-91.1; AUROC 0.919) and 85.6% with CURIAL-Lab (81.6-88.9; AUROC 0.925). 520 patients were prospectively enrolled for point-of-care FBC analysis between February 18, 2021 and May 10, 2021, of whom 436 received confirmatory PCR testing within routine care and 10 (2.3%) tested positive. Median time from patient arrival to availability of CURIAL-Rapide result was 45:00 min (32-64), 16 minutes (26.3%) sooner than LFD results (61:00 min, 37-99; log-rank p<0.0001), and 6:52 h (90.2%) sooner than PCR results (7:37 h, 6:05-15:39; p<0.0001). Sensitivity and specificity of CURIAL-Rapide were 87.5% (52.9-97.8) and 85.4% (81.3-88.7), therefore achieving high NPV (99.7%, 98.2-99.9). CURIAL-Rapide correctly excluded COVID-19 for 58.5% of negative patients who were triaged by a clinician to ‘COVID-19-suspected’ (amber) areas.Impact CURIAL-Lab & CURIAL-Rapide are generalisable, high-throughput screening tests for COVID-19, rapidly excluding the illness with higher NPV than LFDs. CURIAL-Rapide can be used in combination with near-patient FBC analysis for rapid, lab-free screening, and may reduce the number of COVID-19-negative patients triaged to enhanced precautions (‘amber’) clinical areas.Competing Interest StatementDWE reports personal fees from Gilead, outside the submitted work; DAC reports personal fees from Oxford University Innovation, personal fees from BioBeats, personal fees from Sensyne Health, outside the submitted work. No other authors report any conflicts of interest.Funding StatementThis study was supported by the Wellcome Trust/University of Oxford Medical & Life Sciences Translational Fund (Award: 0009350) and the Oxford National Institute of Research (NIHR) Biomedical Research Campus (BRC). The funders of the study had no role in study design, data collection, data analysis, data interpretation, or writing of the manuscript. AS is an NIHR Academic Clinical Fellow. DWE is a Robertson Foundation Fellow and an NIHR Oxford Biomedical Research Centre Senior Fellow. The views expressed are those of the authors and not necessarily those of the NHS, NIHR, or the Wellcome Trust.Author DeclarationsI confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.YesThe details of the IRB/oversight body that provided approval or exemption for the research described are given below:NHS Health Research Authority (HRA) approval was granted for the use of routine clinical and microbiology data from Electronic Health Records for development and validation of artificial intelligence models to detect Covid-19 (CURIAL; NHS HRA IRAS ID 281832).I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.YesI understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).YesI have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.YesData from OUH studied here are available from the Infections in Oxfordshire Research Database (https://oxfordbrc.nihr.ac.uk/research-themes-overview/antimicrobial-resistance-and-modernising-microbiology/infections-in-oxfordshire-research-database-iord/), subject to an application meeting the ethical and governance requirements of the Database. Data from UHB, PUH and BH are available on reasonable request to the respective trusts, subject to HRA requirements. Code and supplementary information for this paper are available online alongside publication.AIArtificial IntelligenceAUROCArea under receiver operating characteristic curveBHBedfordshire Hospitals NHS Foundation TrustCOVID-19Coronavirus Disease 2019CRPC-Reactive ProteinEDEmergency DepartmentEHRElectronic Health RecordsFBCFull Blood CountLFTLiver Function TestLFDRapid antigen lateral flow deviceNHSNational Health ServiceNPVNegative Predictive ValueOUHOxford University Hospitals NHS Foundation TrustPCRReal Time Polymerase Chain ReactionPOCTPoint of Care TestPPVPositive Predictive ValuePUHPortsmouth University Hospitals NHS TrustSARS-CoV-2Severe Acute Respiratory Syndrome Coronavirus 2SDStandard DeviationU&EsUrea, Creatinine and ElectrolytesUHBUniversity Hospitals Birmingham NHS Foundation Trust ER -