Skip to main content
medRxiv
  • Home
  • About
  • Submit
  • ALERTS / RSS
Advanced Search

Benchmark of thirteen bioinformatic pipelines for metagenomic virus diagnostics using datasets from clinical samples

View ORCID ProfileJutte J.C. de Vries, Julianne R. Brown, Nicole Fischer, Igor A. Sidorov, Sofia Morfopoulou, Jiabin Huang, Bas B. Oude Munnink, Arzu Sayiner, Alihan Bulgurcu, Christophe Rodriguez, Guillaume Gricourt, Els Keyaerts, Leen Beller, Claudia Bachofen, Jakub Kubacki, Samuel Cordey, Florian Laubscher, Dennis Schmitz, Martin Beer, Dirk Hoeper, Michael Huber, Verena Kufner, Maryam Zaheri, Aitana Lebrand, Anna Papa, Sander van Boheemen, Aloys C.M. Kroes, Judith Breuer, F. Xavier Lopez-Labrador, Eric C.J. Claas
doi: https://doi.org/10.1101/2021.05.04.21256618
Jutte J.C. de Vries
1Clinical Microbiological Laboratory, department of Medical Microbiology, Leiden University Medical Center, Leiden, the Netherlands
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Jutte J.C. de Vries
  • For correspondence: jjcdevries@lumc.nl
Julianne R. Brown
2Microbiology, Virology and Infection Prevention & Control, Great Ormond Street Hospital for Children NHS Foundation Trust, London, United Kingdom
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Nicole Fischer
3University Medical Center Hamburg-Eppendorf, UKE Institute for Medical Microbiology, Virology and Hygiene, Germany
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Igor A. Sidorov
1Clinical Microbiological Laboratory, department of Medical Microbiology, Leiden University Medical Center, Leiden, the Netherlands
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Sofia Morfopoulou
2aDivision of Infection and Immunity, University College London, London, United Kingdom
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Jiabin Huang
3University Medical Center Hamburg-Eppendorf, UKE Institute for Medical Microbiology, Virology and Hygiene, Germany
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Bas B. Oude Munnink
4Viroscience, Erasmus Medical Center, Rotterdam, the Netherlands
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Arzu Sayiner
5Dokuz Eylul University, Medical Faculty, Izmir, Turkey
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Alihan Bulgurcu
5Dokuz Eylul University, Medical Faculty, Izmir, Turkey
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Christophe Rodriguez
6Hospital Henri Mondor, Paris, France
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Guillaume Gricourt
6Hospital Henri Mondor, Paris, France
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Els Keyaerts
7Laboratory of Clinical and Epidemiological Virology (Rega Institute), KU Leuven, Belgium
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Leen Beller
7Laboratory of Clinical and Epidemiological Virology (Rega Institute), KU Leuven, Belgium
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Claudia Bachofen
8Institute of Virology, University of Zurich, Switzerland
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Jakub Kubacki
8Institute of Virology, University of Zurich, Switzerland
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Samuel Cordey
9Laboratory of Virology, University Hospitals of Geneva, Geneva, Switzerland
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Florian Laubscher
9Laboratory of Virology, University Hospitals of Geneva, Geneva, Switzerland
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Dennis Schmitz
10RIVM National Institute for Public Health and Environment, Bilthoven, the Netherlands
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Martin Beer
11Friedrich-Loeffler-Institute, Institute of Diagnostic Virology, Greifswald, Germany
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Dirk Hoeper
11Friedrich-Loeffler-Institute, Institute of Diagnostic Virology, Greifswald, Germany
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Michael Huber
12Institute of Medical Virology, University of Zurich, Switzerland
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Verena Kufner
12Institute of Medical Virology, University of Zurich, Switzerland
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Maryam Zaheri
12Institute of Medical Virology, University of Zurich, Switzerland
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Aitana Lebrand
13Swiss Institute of Bioinformatics, Geneva, Switzerland
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Anna Papa
14Department of Microbiology, Medical School, Aristotle University of Thessaloniki, Greece
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Sander van Boheemen
4Viroscience, Erasmus Medical Center, Rotterdam, the Netherlands
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Aloys C.M. Kroes
1Clinical Microbiological Laboratory, department of Medical Microbiology, Leiden University Medical Center, Leiden, the Netherlands
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Judith Breuer
2Microbiology, Virology and Infection Prevention & Control, Great Ormond Street Hospital for Children NHS Foundation Trust, London, United Kingdom
2aDivision of Infection and Immunity, University College London, London, United Kingdom
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
F. Xavier Lopez-Labrador
15Virology Laboratory, Genomics and Health Area, Center for Public Health Research (FISABIO-Public Health), Generalitat Valenciana and Microbiology & Ecology Department, University of Valencia, Spain
16CIBERESP, Instituto de Salud Carlos III, Spain
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Eric C.J. Claas
1Clinical Microbiological Laboratory, department of Medical Microbiology, Leiden University Medical Center, Leiden, the Netherlands
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • Abstract
  • Full Text
  • Info/History
  • Metrics
  • Supplementary material
  • Data/Code
  • Preview PDF
Loading

Abstract

Metagenomic sequencing is increasingly being used in clinical settings for difficult to diagnose cases. The performance of viral metagenomic protocols relies to a large extent on the bioinformatic analysis. In this study, the European Society for Clinical Virology (ESCV) Network on NGS (ENNGS) initiated a benchmark of metagenomic pipelines currently used in clinical virological laboratories.

Methods Metagenomic datasets from 13 clinical samples from patients with encephalitis or viral respiratory infections characterized by PCR were selected. The datasets were analysed with 13 different pipelines currently used in virological diagnostic laboratories of participating ENNGS members. The pipelines and classification tools were: Centrifuge, DAMIAN, DIAMOND, DNASTAR, FEVIR, Genome Detective, Jovian, MetaMIC, MetaMix, One Codex, RIEMS, VirMet, and Taxonomer. Performance, characteristics, clinical use, and user-friendliness of these pipelines were analysed.

Results Overall, viral pathogens with high loads were detected by all the evaluated metagenomic pipelines. In contrast, lower abundance pathogens and mixed infections were only detected by 3/13 pipelines, namely DNASTAR, FEVIR, and MetaMix. Overall sensitivity ranged from 80% (10/13) to 100% (13/13 datasets). Overall positive predictive value ranged from 71-100%. The majority of the pipelines classified sequences based on nucleotide similarity (8/13), only a minority used amino acid similarity, and 6 of the 13 pipelines assembled sequences de novo. No clear differences in performance were detected that correlated with these classification approaches. Read counts of target viruses varied between the pipelines over a range of 2-3 log, indicating differences in limit of detection.

Conclusion A wide variety of viral metagenomic pipelines is currently used in the participating clinical diagnostic laboratories. Detection of low abundant viral pathogens and mixed infections remains a challenge, implicating the need for standardization and validation of metagenomic analysis for clinical diagnostic use. Future studies should address the selective effects due to the choice of different reference viral databases.

Competing Interest Statement

The authors have declared no competing interest.

Funding Statement

MH was supported by the Clinical Research Priority Program 'Comprehensive Genomic Pathogen Detection' of the University of Zurich FXL receives funding from Instituto de Salud Carlos III, Spain (Grant numbers PI18/01824 and PI18/01759 and CIBEResp).

Author Declarations

I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.

Yes

The details of the IRB/oversight body that provided approval or exemption for the research described are given below:

This is a comparison of pipelines using datasets. No clinical intervention was made.

All necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived.

Yes

I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).

Yes

I have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.

Yes

Footnotes

  • jjcdevries{at}lumc.nl, I.A.Sidorov{at}lumc.nl, A.C.M.Kroes{at}lumc.nl, E.C.J.Claas{at}lumc.nl, julianne.brown{at}gosh.nhs.uk, breuej{at}gosh.nhs.uk, sofia.morfopoulou.10{at}ucl.ac.uk, nfischer{at}uke.de, j.huang{at}uke.de, b.oudemunnink{at}erasmusmc.nl, s.vanboheemen{at}erasmusmc.nl, arzu.sayiner{at}deu.edu.tr, alihanbulgurcu{at}gmail.com, christophe.rodriguez{at}aphp.fr, guillaume.gricourt{at}aphp.fr, els.keyaerts{at}kuleuven.be, leen.beller{at}kuleuven.be, claudia.bachofen{at}uzh.ch, jakub.kubacki{at}uzh.ch, Samuel.Cordey{at}hcuge.ch, Florian.Laubscher{at}hcuge.ch, Dennis.Schmitz{at}RIVM.nl, martin.beer{at}fli.de, dirk.hoeper{at}fli.de, huber.michael{at}virology.uzh.ch, kufner.verena{at}virology.uzh.ch, zaheri.maryam{at}virology.uzh.ch, aitana.lebrand{at}sib.swiss, annap{at}auth.gr, F.Xavier.Lopez{at}uv.es

Data Availability

The FASTQ datasets were and remain publicly available for user-friendly downloading at https://veb.lumc.nl/CliniMG (hosted by the dept. MM, LUMC, Leiden), and part of the datasets were additionally accessible via a COMPARE Data Hub at http://www.ebi.ac.uk/ena/pathogens (hosted by the European Bioinformatics Institute, EMBL-EBI)

https://veb.lumc.nl/CliniMG

Copyright 
The copyright holder for this preprint is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. It is made available under a CC-BY-NC-ND 4.0 International license.
Back to top
PreviousNext
Posted May 08, 2021.
Download PDF

Supplementary Material

Data/Code
Email

Thank you for your interest in spreading the word about medRxiv.

NOTE: Your email address is requested solely to identify you as the sender of this article.

Enter multiple addresses on separate lines or separate them with commas.
Benchmark of thirteen bioinformatic pipelines for metagenomic virus diagnostics using datasets from clinical samples
(Your Name) has forwarded a page to you from medRxiv
(Your Name) thought you would like to see this page from the medRxiv website.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.
Share
Benchmark of thirteen bioinformatic pipelines for metagenomic virus diagnostics using datasets from clinical samples
Jutte J.C. de Vries, Julianne R. Brown, Nicole Fischer, Igor A. Sidorov, Sofia Morfopoulou, Jiabin Huang, Bas B. Oude Munnink, Arzu Sayiner, Alihan Bulgurcu, Christophe Rodriguez, Guillaume Gricourt, Els Keyaerts, Leen Beller, Claudia Bachofen, Jakub Kubacki, Samuel Cordey, Florian Laubscher, Dennis Schmitz, Martin Beer, Dirk Hoeper, Michael Huber, Verena Kufner, Maryam Zaheri, Aitana Lebrand, Anna Papa, Sander van Boheemen, Aloys C.M. Kroes, Judith Breuer, F. Xavier Lopez-Labrador, Eric C.J. Claas
medRxiv 2021.05.04.21256618; doi: https://doi.org/10.1101/2021.05.04.21256618
Digg logo Reddit logo Twitter logo Facebook logo Google logo LinkedIn logo Mendeley logo
Citation Tools
Benchmark of thirteen bioinformatic pipelines for metagenomic virus diagnostics using datasets from clinical samples
Jutte J.C. de Vries, Julianne R. Brown, Nicole Fischer, Igor A. Sidorov, Sofia Morfopoulou, Jiabin Huang, Bas B. Oude Munnink, Arzu Sayiner, Alihan Bulgurcu, Christophe Rodriguez, Guillaume Gricourt, Els Keyaerts, Leen Beller, Claudia Bachofen, Jakub Kubacki, Samuel Cordey, Florian Laubscher, Dennis Schmitz, Martin Beer, Dirk Hoeper, Michael Huber, Verena Kufner, Maryam Zaheri, Aitana Lebrand, Anna Papa, Sander van Boheemen, Aloys C.M. Kroes, Judith Breuer, F. Xavier Lopez-Labrador, Eric C.J. Claas
medRxiv 2021.05.04.21256618; doi: https://doi.org/10.1101/2021.05.04.21256618

Citation Manager Formats

  • BibTeX
  • Bookends
  • EasyBib
  • EndNote (tagged)
  • EndNote 8 (xml)
  • Medlars
  • Mendeley
  • Papers
  • RefWorks Tagged
  • Ref Manager
  • RIS
  • Zotero
  • Tweet Widget
  • Facebook Like
  • Google Plus One

Subject Area

  • Infectious Diseases (except HIV/AIDS)
Subject Areas
All Articles
  • Addiction Medicine (161)
  • Allergy and Immunology (414)
  • Anesthesia (90)
  • Cardiovascular Medicine (857)
  • Dentistry and Oral Medicine (159)
  • Dermatology (97)
  • Emergency Medicine (248)
  • Endocrinology (including Diabetes Mellitus and Metabolic Disease) (393)
  • Epidemiology (8557)
  • Forensic Medicine (4)
  • Gastroenterology (383)
  • Genetic and Genomic Medicine (1749)
  • Geriatric Medicine (167)
  • Health Economics (372)
  • Health Informatics (1239)
  • Health Policy (620)
  • Health Systems and Quality Improvement (467)
  • Hematology (196)
  • HIV/AIDS (372)
  • Infectious Diseases (except HIV/AIDS) (10292)
  • Intensive Care and Critical Care Medicine (553)
  • Medical Education (192)
  • Medical Ethics (51)
  • Nephrology (211)
  • Neurology (1676)
  • Nursing (97)
  • Nutrition (249)
  • Obstetrics and Gynecology (326)
  • Occupational and Environmental Health (450)
  • Oncology (928)
  • Ophthalmology (263)
  • Orthopedics (101)
  • Otolaryngology (172)
  • Pain Medicine (112)
  • Palliative Medicine (40)
  • Pathology (252)
  • Pediatrics (534)
  • Pharmacology and Therapeutics (248)
  • Primary Care Research (207)
  • Psychiatry and Clinical Psychology (1765)
  • Public and Global Health (3835)
  • Radiology and Imaging (623)
  • Rehabilitation Medicine and Physical Therapy (320)
  • Respiratory Medicine (520)
  • Rheumatology (208)
  • Sexual and Reproductive Health (166)
  • Sports Medicine (158)
  • Surgery (190)
  • Toxicology (36)
  • Transplantation (101)
  • Urology (76)