Abstract
Objectives The aims of this study are firstly to investigate the diagnostic and triage performance of symptom checkers, secondly to assess their potential impact on healthcare utilisation and thirdly to investigate for variation in performance between systems.
Setting Publicly available symptom checkers
Participants Publicly available symptom-checkers were identified. A standardised set of 50 clinical vignettes was developed and systematically run through each system by a non-clinical researcher.
Primary and secondary outcome measures System accuracy was assessed by measuring the percentage of times the correct diagnosis was a) listed first, b) within the top five diagnoses listed and c) listed at all. The safety of the disposition advice was assessed by comparing it with national guidelines for each vignette.
Results Twelve tools were identified and included. Mean diagnostic accuracy of the systems was poor, with the correct diagnosis being listed first on 37.7% (Range 22.2 to 72.0%) of occasions and present in the top five diagnoses on 51.0% (Range 22.2 to 84.0%). 51.0% of systems suggested additional resource utilisation above that recommended by national guidelines (range 18.0% to 61.2%). Both diagnostic accuracy and appropriate resource recommendation varied substantially between systems.
Conclusions There is wide variation in performance between available symptom checkers and overall performance is significantly below what would be accepted in any other medical field, though some do achieve a good level of accuracy and safety of disposition. External validation and regulation are urgently required to ensure these public facing tools are safe.
Strengths and Limitations
Data collection was undertaken by non-clinically trained staff to replicate patient behaviour and there was random sampling to test the inter-rater reliability
Clinical vignettes were agreed by a clinical team consisting of a GP, a pharmacist and a hospital emergency care consultant
Current UK guidelines were used to assess service utilisation. Where symptom checkers were developed outside of the UK the disposition advice may be unlikely to be aligned due to different jurisdictions
This research was a limited indirect study on the variety of terms and language patients might use in their interactions with these systems
There was no assessment of how a clinician would diagnose and triage a patient presenting with the vignette symptoms
Competing Interest Statement
All authors have completed the Unified Competing Interest form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare: Doctorlink employees had sight of the manuscript and were able to make comments on it, but all decisions were made independently by the authorship team. A copy of the draft manuscripts that they viewed, with all suggestions, was attached with the submitted article for transparency purposes.
Funding Statement
This study was fully funded as an external review of the sector by Doctorlink Ltd.
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
The details of the IRB/oversight body that provided approval or exemption for the research described are given below:
This study did not require IRB/ Oversight body approval
All necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.
Yes
Data Availability
The Corresponding Author has the right to grant on behalf of all authors and does grant on behalf of all authors, a worldwide licence to the Publishers and its licensees in perpetuity, in all forms, formats and media (whether known now or created in the future), to i) publish, reproduce, distribute, display and store the Contribution, ii) translate the Contribution into other languages, create adaptations, reprints, include within collections and create summaries, extracts and/or, abstracts of the Contribution, iii) create any other derivative work(s) based on the Contribution, iv) to exploit all subsidiary rights in the Contribution, v) the inclusion of electronic links from the Contribution to third party material where-ever it may be located; and, vi) licence any third party to do any or all of the above.
