PT - JOURNAL ARTICLE AU - Jutte J.C. de Vries AU - Julianne R. Brown AU - Nicole Fischer AU - Igor A. Sidorov AU - Sofia Morfopoulou AU - Jiabin Huang AU - Bas B. Oude Munnink AU - Arzu Sayiner AU - Alihan Bulgurcu AU - Christophe Rodriguez AU - Guillaume Gricourt AU - Els Keyaerts AU - Leen Beller AU - Claudia Bachofen AU - Jakub Kubacki AU - Samuel Cordey AU - Florian Laubscher AU - Dennis Schmitz AU - Martin Beer AU - Dirk Hoeper AU - Michael Huber AU - Verena Kufner AU - Maryam Zaheri AU - Aitana Lebrand AU - Anna Papa AU - Sander van Boheemen AU - Aloys C.M. Kroes AU - Judith Breuer AU - F. Xavier Lopez-Labrador AU - Eric C.J. Claas AU - on behalf of the ESCV Network on Next-Generation Sequencing TI - Benchmark of thirteen bioinformatic pipelines for metagenomic virus diagnostics using datasets from clinical samples AID - 10.1101/2021.05.04.21256618 DP - 2021 Jan 01 TA - medRxiv PG - 2021.05.04.21256618 4099 - http://medrxiv.org/content/early/2021/05/08/2021.05.04.21256618.short 4100 - http://medrxiv.org/content/early/2021/05/08/2021.05.04.21256618.full AB - Metagenomic sequencing is increasingly being used in clinical settings for difficult to diagnose cases. The performance of viral metagenomic protocols relies to a large extent on the bioinformatic analysis. In this study, the European Society for Clinical Virology (ESCV) Network on NGS (ENNGS) initiated a benchmark of metagenomic pipelines currently used in clinical virological laboratories.Methods Metagenomic datasets from 13 clinical samples from patients with encephalitis or viral respiratory infections characterized by PCR were selected. The datasets were analysed with 13 different pipelines currently used in virological diagnostic laboratories of participating ENNGS members. The pipelines and classification tools were: Centrifuge, DAMIAN, DIAMOND, DNASTAR, FEVIR, Genome Detective, Jovian, MetaMIC, MetaMix, One Codex, RIEMS, VirMet, and Taxonomer. Performance, characteristics, clinical use, and user-friendliness of these pipelines were analysed.Results Overall, viral pathogens with high loads were detected by all the evaluated metagenomic pipelines. In contrast, lower abundance pathogens and mixed infections were only detected by 3/13 pipelines, namely DNASTAR, FEVIR, and MetaMix. Overall sensitivity ranged from 80% (10/13) to 100% (13/13 datasets). Overall positive predictive value ranged from 71-100%. The majority of the pipelines classified sequences based on nucleotide similarity (8/13), only a minority used amino acid similarity, and 6 of the 13 pipelines assembled sequences de novo. No clear differences in performance were detected that correlated with these classification approaches. Read counts of target viruses varied between the pipelines over a range of 2-3 log, indicating differences in limit of detection.Conclusion A wide variety of viral metagenomic pipelines is currently used in the participating clinical diagnostic laboratories. Detection of low abundant viral pathogens and mixed infections remains a challenge, implicating the need for standardization and validation of metagenomic analysis for clinical diagnostic use. Future studies should address the selective effects due to the choice of different reference viral databases.Competing Interest StatementThe authors have declared no competing interest.Funding StatementMH was supported by the Clinical Research Priority Program 'Comprehensive Genomic Pathogen Detection' of the University of Zurich FXL receives funding from Instituto de Salud Carlos III, Spain (Grant numbers PI18/01824 and PI18/01759 and CIBEResp). Author DeclarationsI confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.YesThe details of the IRB/oversight body that provided approval or exemption for the research described are given below:This is a comparison of pipelines using datasets. No clinical intervention was made.All necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived.YesI understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).YesI have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.YesThe FASTQ datasets were and remain publicly available for user-friendly downloading at https://veb.lumc.nl/CliniMG (hosted by the dept. MM, LUMC, Leiden), and part of the datasets were additionally accessible via a COMPARE Data Hub at http://www.ebi.ac.uk/ena/pathogens (hosted by the European Bioinformatics Institute, EMBL-EBI) https://veb.lumc.nl/CliniMG