PT - JOURNAL ARTICLE AU - Ceney, Adam AU - Tolond, Stephanie AU - Glowinski, Andrzej AU - Marks, Ben AU - Swift, Simon AU - Palser, Tom TI - Accuracy of online symptom checkers and the potential impact on service utilisation AID - 10.1101/2020.07.07.20147975 DP - 2020 Jan 01 TA - medRxiv PG - 2020.07.07.20147975 4099 - http://medrxiv.org/content/early/2020/07/08/2020.07.07.20147975.short 4100 - http://medrxiv.org/content/early/2020/07/08/2020.07.07.20147975.full AB - Objectives The aims of this study are firstly to investigate the diagnostic and triage performance of symptom checkers, secondly to assess their potential impact on healthcare utilisation and thirdly to investigate for variation in performance between systems.Setting Publicly available symptom checkersParticipants Publicly available symptom-checkers were identified. A standardised set of 50 clinical vignettes was developed and systematically run through each system by a non-clinical researcher.Primary and secondary outcome measures System accuracy was assessed by measuring the percentage of times the correct diagnosis was a) listed first, b) within the top five diagnoses listed and c) listed at all. The safety of the disposition advice was assessed by comparing it with national guidelines for each vignette.Results Twelve tools were identified and included. Mean diagnostic accuracy of the systems was poor, with the correct diagnosis being listed first on 37.7% (Range 22.2 to 72.0%) of occasions and present in the top five diagnoses on 51.0% (Range 22.2 to 84.0%). 51.0% of systems suggested additional resource utilisation above that recommended by national guidelines (range 18.0% to 61.2%). Both diagnostic accuracy and appropriate resource recommendation varied substantially between systems.Conclusions There is wide variation in performance between available symptom checkers and overall performance is significantly below what would be accepted in any other medical field, though some do achieve a good level of accuracy and safety of disposition. External validation and regulation are urgently required to ensure these public facing tools are safe.Strengths and LimitationsData collection was undertaken by non-clinically trained staff to replicate patient behaviour and there was random sampling to test the inter-rater reliabilityClinical vignettes were agreed by a clinical team consisting of a GP, a pharmacist and a hospital emergency care consultantCurrent UK guidelines were used to assess service utilisation. Where symptom checkers were developed outside of the UK the disposition advice may be unlikely to be aligned due to different jurisdictionsThis research was a limited indirect study on the variety of terms and language patients might use in their interactions with these systemsThere was no assessment of how a clinician would diagnose and triage a patient presenting with the vignette symptomsCompeting Interest StatementAll authors have completed the Unified Competing Interest form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare: Doctorlink employees had sight of the manuscript and were able to make comments on it, but all decisions were made independently by the authorship team. A copy of the draft manuscripts that they viewed, with all suggestions, was attached with the submitted article for transparency purposes. Funding StatementThis study was fully funded as an external review of the sector by Doctorlink Ltd.Author DeclarationsI confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.YesThe details of the IRB/oversight body that provided approval or exemption for the research described are given below:This study did not require IRB/ Oversight body approvalAll necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived.YesI understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).YesI have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.YesThe Corresponding Author has the right to grant on behalf of all authors and does grant on behalf of all authors, a worldwide licence to the Publishers and its licensees in perpetuity, in all forms, formats and media (whether known now or created in the future), to i) publish, reproduce, distribute, display and store the Contribution, ii) translate the Contribution into other languages, create adaptations, reprints, include within collections and create summaries, extracts and/or, abstracts of the Contribution, iii) create any other derivative work(s) based on the Contribution, iv) to exploit all subsidiary rights in the Contribution, v) the inclusion of electronic links from the Contribution to third party material where-ever it may be located; and, vi) licence any third party to do any or all of the above.