Abstract
Importance Access to human expertise for affordable and efficient triage of ophthalmic conditions is inconsistent. With recent advancements in publicly available artificial intelligence (AI) chatbots, individuals may turn to these tools for triage of ophthalmic complaints.
Objective To evaluate the triage performance of AI chatbots for ophthalmic conditions
Design Cross-sectional study
Setting Single center
Participants Ophthalmology trainees, OpenAI ChatGPT (GPT-4), Bing Chat, and WebMD Symptom Checker
Methods Forty-four clinical vignettes representing common ophthalmic complaints were developed, and a standardized pathway of prompts was presented to each tool in March 2023.
Primary outcomes were proportion of responses with correct diagnosis listed in the top three possible diagnoses and proportion with correct triage urgency. Ancillary outcomes included presence of grossly inaccurate statements, mean reading grade level, mean response word count, proportion with attribution, most common sources cited, and proportion with a disclaimer regarding chatbot limitations in dispensing medical advice.
Results The physician respondents, ChatGPT, Bing Chat, and WebMD listed the appropriate diagnosis among the top three suggestions in 42 (95%), 41 (93%), 34 (77%), and 8 (33%) cases, respectively. Triage urgency was appropriate in 38 (86%), 43 (98%), and 37 (84%) cases for the physicians, ChatGPT, and Bing Chat, correspondingly.
Conclusions and Relevance ChatGPT using the GPT-4 model offered high diagnostic and triage accuracy that was comparable to the physician respondents, with no grossly inaccurate statements. Bing Chat had lower accuracy, some instances of grossly inaccurate statements, and a tendency to overestimate triage urgency.
Competing Interest Statement
The authors have declared no competing interest.
Funding Statement
Details of funding received: -Foundation Fighting Blindness Career Development Award CD-C-0918-0748-EEC (NJ) -National Eye Insitute P30EY006360 (NJ)
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
The details of the IRB/oversight body that provided approval or exemption for the research described are given below:
This cross-sectional study was determined exempt from formal review by the Emory University Institutional Review Board
I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.
Yes
Data Availability
All data produced in the present work are contained in the manuscript and supplemental materials
