Skip to main content
medRxiv
  • Home
  • About
  • Submit
  • ALERTS / RSS
Advanced Search

Accuracy of online symptom checkers and the potential impact on service utilisation

Adam Ceney, Stephanie Tolond, Andrzej Glowinski, Ben Marks, Simon Swift, Tom Palser
doi: https://doi.org/10.1101/2020.07.07.20147975
Adam Ceney
1Methods Analytics Ltd, Sheffield Digital Campus, Electric Works, Sheffield, S1 2BJ United Kingdom
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • For correspondence: adam.ceney@gmail.com
Stephanie Tolond
1Methods Analytics Ltd, Sheffield Digital Campus, Electric Works, Sheffield, S1 2BJ United Kingdom
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Andrzej Glowinski
1Methods Analytics Ltd, Sheffield Digital Campus, Electric Works, Sheffield, S1 2BJ United Kingdom
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Ben Marks
1Methods Analytics Ltd, Sheffield Digital Campus, Electric Works, Sheffield, S1 2BJ United Kingdom
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Simon Swift
1Methods Analytics Ltd, Sheffield Digital Campus, Electric Works, Sheffield, S1 2BJ United Kingdom
4University of Exeter Business School (INDEX)
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Tom Palser
1Methods Analytics Ltd, Sheffield Digital Campus, Electric Works, Sheffield, S1 2BJ United Kingdom
2Department of Surgery, University Hospitals of Leicester NHS Trust
3SAPPHIRE, Department of Health Sciences, University of Leicester, Leicester, UK
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • Abstract
  • Full Text
  • Info/History
  • Metrics
  • Supplementary material
  • Data/Code
  • Preview PDF
Loading

Abstract

Objectives The aims of this study are firstly to investigate the diagnostic and triage performance of symptom checkers, secondly to assess their potential impact on healthcare utilisation and thirdly to investigate for variation in performance between systems.

Setting Publicly available symptom checkers

Participants Publicly available symptom-checkers were identified. A standardised set of 50 clinical vignettes was developed and systematically run through each system by a non-clinical researcher.

Primary and secondary outcome measures System accuracy was assessed by measuring the percentage of times the correct diagnosis was a) listed first, b) within the top five diagnoses listed and c) listed at all. The safety of the disposition advice was assessed by comparing it with national guidelines for each vignette.

Results Twelve tools were identified and included. Mean diagnostic accuracy of the systems was poor, with the correct diagnosis being listed first on 37.7% (Range 22.2 to 72.0%) of occasions and present in the top five diagnoses on 51.0% (Range 22.2 to 84.0%). 51.0% of systems suggested additional resource utilisation above that recommended by national guidelines (range 18.0% to 61.2%). Both diagnostic accuracy and appropriate resource recommendation varied substantially between systems.

Conclusions There is wide variation in performance between available symptom checkers and overall performance is significantly below what would be accepted in any other medical field, though some do achieve a good level of accuracy and safety of disposition. External validation and regulation are urgently required to ensure these public facing tools are safe.

Strengths and Limitations

  • Data collection was undertaken by non-clinically trained staff to replicate patient behaviour and there was random sampling to test the inter-rater reliability

  • Clinical vignettes were agreed by a clinical team consisting of a GP, a pharmacist and a hospital emergency care consultant

  • Current UK guidelines were used to assess service utilisation. Where symptom checkers were developed outside of the UK the disposition advice may be unlikely to be aligned due to different jurisdictions

  • This research was a limited indirect study on the variety of terms and language patients might use in their interactions with these systems

  • There was no assessment of how a clinician would diagnose and triage a patient presenting with the vignette symptoms

Competing Interest Statement

All authors have completed the Unified Competing Interest form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare: Doctorlink employees had sight of the manuscript and were able to make comments on it, but all decisions were made independently by the authorship team. A copy of the draft manuscripts that they viewed, with all suggestions, was attached with the submitted article for transparency purposes.

Funding Statement

This study was fully funded as an external review of the sector by Doctorlink Ltd.

Author Declarations

I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.

Yes

The details of the IRB/oversight body that provided approval or exemption for the research described are given below:

This study did not require IRB/ Oversight body approval

All necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived.

Yes

I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).

Yes

I have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.

Yes

Data Availability

The Corresponding Author has the right to grant on behalf of all authors and does grant on behalf of all authors, a worldwide licence to the Publishers and its licensees in perpetuity, in all forms, formats and media (whether known now or created in the future), to i) publish, reproduce, distribute, display and store the Contribution, ii) translate the Contribution into other languages, create adaptations, reprints, include within collections and create summaries, extracts and/or, abstracts of the Contribution, iii) create any other derivative work(s) based on the Contribution, iv) to exploit all subsidiary rights in the Contribution, v) the inclusion of electronic links from the Contribution to third party material where-ever it may be located; and, vi) licence any third party to do any or all of the above.

Copyright 
The copyright holder for this preprint is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. All rights reserved. No reuse allowed without permission.
Back to top
PreviousNext
Posted July 08, 2020.
Download PDF

Supplementary Material

Data/Code
Email

Thank you for your interest in spreading the word about medRxiv.

NOTE: Your email address is requested solely to identify you as the sender of this article.

Enter multiple addresses on separate lines or separate them with commas.
Accuracy of online symptom checkers and the potential impact on service utilisation
(Your Name) has forwarded a page to you from medRxiv
(Your Name) thought you would like to see this page from the medRxiv website.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.
Share
Accuracy of online symptom checkers and the potential impact on service utilisation
Adam Ceney, Stephanie Tolond, Andrzej Glowinski, Ben Marks, Simon Swift, Tom Palser
medRxiv 2020.07.07.20147975; doi: https://doi.org/10.1101/2020.07.07.20147975
Digg logo Reddit logo Twitter logo CiteULike logo Facebook logo Google logo Mendeley logo
Citation Tools
Accuracy of online symptom checkers and the potential impact on service utilisation
Adam Ceney, Stephanie Tolond, Andrzej Glowinski, Ben Marks, Simon Swift, Tom Palser
medRxiv 2020.07.07.20147975; doi: https://doi.org/10.1101/2020.07.07.20147975

Citation Manager Formats

  • BibTeX
  • Bookends
  • EasyBib
  • EndNote (tagged)
  • EndNote 8 (xml)
  • Medlars
  • Mendeley
  • Papers
  • RefWorks Tagged
  • Ref Manager
  • RIS
  • Zotero
  • Tweet Widget
  • Facebook Like
  • Google Plus One

Subject Area

  • Health Systems and Quality Improvement
Subject Areas
All Articles
  • Addiction Medicine (62)
  • Allergy and Immunology (142)
  • Anesthesia (46)
  • Cardiovascular Medicine (412)
  • Dentistry and Oral Medicine (69)
  • Dermatology (47)
  • Emergency Medicine (142)
  • Endocrinology (including Diabetes Mellitus and Metabolic Disease) (171)
  • Epidemiology (4843)
  • Forensic Medicine (3)
  • Gastroenterology (183)
  • Genetic and Genomic Medicine (674)
  • Geriatric Medicine (70)
  • Health Economics (192)
  • Health Informatics (626)
  • Health Policy (318)
  • Health Systems and Quality Improvement (203)
  • Hematology (85)
  • HIV/AIDS (156)
  • Infectious Diseases (except HIV/AIDS) (5326)
  • Intensive Care and Critical Care Medicine (328)
  • Medical Education (93)
  • Medical Ethics (25)
  • Nephrology (75)
  • Neurology (685)
  • Nursing (42)
  • Nutrition (114)
  • Obstetrics and Gynecology (126)
  • Occupational and Environmental Health (205)
  • Oncology (439)
  • Ophthalmology (140)
  • Orthopedics (36)
  • Otolaryngology (89)
  • Pain Medicine (35)
  • Palliative Medicine (16)
  • Pathology (129)
  • Pediatrics (194)
  • Pharmacology and Therapeutics (131)
  • Primary Care Research (84)
  • Psychiatry and Clinical Psychology (778)
  • Public and Global Health (1810)
  • Radiology and Imaging (323)
  • Rehabilitation Medicine and Physical Therapy (138)
  • Respiratory Medicine (255)
  • Rheumatology (86)
  • Sexual and Reproductive Health (69)
  • Sports Medicine (62)
  • Surgery (100)
  • Toxicology (23)
  • Transplantation (29)
  • Urology (37)