ABSTRACT
Purpose To evaluate the efficiency of large language models (LLMs) including ChatGPT to assist in diagnosing neuro-ophthalmic diseases based on case reports.
Design Prospective study
Subjects or Participants We selected 22 different case reports of neuro-ophthalmic diseases from a publicly available online database. These cases included a wide range of chronic and acute diseases that are commonly seen by neuro-ophthalmic sub-specialists.
Methods We inserted the text from each case as a new prompt into both ChatGPT v3.5 and ChatGPT Plus v4.0 and asked for the most probable diagnosis. We then presented the exact information to two neuro-ophthalmologists and recorded their diagnoses followed by comparison to responses from both versions of ChatGPT.
Main Outcome Measures Diagnostic accuracy in terms of number of correctly diagnosed cases among diagnoses.
Results ChatGPT v3.5, ChatGPT Plus v4.0, and the two neuro-ophthalmologists were correct in 13 (59%), 18 (82%), 19 (86%), and 19 (86%) out of 22 cases, respectively. The agreement between the various diagnostic sources were as follows: ChatGPT v3.5 and ChatGPT Plus v4.0, 13 (59%); ChatGPT v3.5 and the first neuro-ophthalmologist, 12 (55%); ChatGPT v3.5 and the second neuro-ophthalmologist, 12 (55%); ChatGPT Plus v4.0 and the first neuro-ophthalmologist, 17 (77%); ChatGPT Plus v4.0 and the second neuro-ophthalmologist, 16 (73%); and first and second neuro-ophthalmologists 17 (17%).
Conclusions The accuracy of ChatGPT v3.5 and ChatGPT Plus v4.0 in diagnosing patients with neuro-ophthalmic diseases was 59% and 82%, respectively. With further development, ChatGPT Plus v4.0 may have potential to be used in clinical care settings to assist clinicians in providing quick, accurate diagnoses of patients in neuro-ophthalmology. The applicability of using LLMs like ChatGPT in clinical settings that lack access to subspeciality trained neuro-ophthalmologists deserves further research.
Summary Highlights
- The goal of this study was to explore the capabilities of ChatGPT for the diagnoses of different neuro-ophthalmic diseases using specific case examples.
- There was general agreement between ChatGPT Plus v4.0 and two neuro-ophthalmologists in final diagnoses.
- ChatGPT was more general while neuro-ophthalmologists were more methodical and specific when listing diagnoses.
Competing Interest Statement
The authors have declared no competing interest.
Funding Statement
This work was supported by NIH Grants R01EY033005 (SY), R21EY031725 (SY), and grants from Research to Prevent Blindness (RPB), New York (SY). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.
Yes
Footnotes
Conflicts of interest/Competing interests: Authors declare no relevant conflict of interest(s) to disclose.
Data Availability
All data produced are available online at the University of Iowa's publicly accessible database (https://webeye.ophth.uiowa.edu/eyeforum/cases.html).
