Abstract
Background Radiological imaging guidelines are crucial for accurate diagnosis and optimal patient care as they result in standardized procedures and thus reduce inappropriate imaging studies. In the present study, we investigated the potential to support clinical decision-making using an interactive chatbot designed to provide personalized imaging recommendations based on indexed and vectorized American College of Radiology (ACR) appropriateness criteria documents.
Methods We utilized 209 ACR appropriateness criteria documents as specialized knowledge base and employed LlamaIndex and the ChatGPT 3.5-Turbo to create an appropriateness criteria contexted chatbot (accGPT). Fifty clinical case files were used to compare the accGPT’s performance against radiologists at varying experience levels and to generic ChatGPT 3.5 and 4.0.
Results All chatbots reached at least human performance level. For the 50 case files, the accGPT provided a median of 83% (95% CI 82-84) ‘usually appropriate’ recommendations, while radiologists provided a median of 66% (95% CI 62-70). GPT 3.5-Turbo 70% (95% CI 67-73) and GPT 4 79% (95% CI 76-81) correct answers. Consistency was highest for the accGPT with almost perfect Fleiss’ Kappa of 0.82. Further, the chatbots provided substantial time and cost savings, with an average decision time of 5 minutes and a cost of 0.19 Euro for all cases, compared to 50 minutes and 29.99 Euro for radiologists (both p < 0.01).
Conclusion ChatGPT-based algorithms have the potential to substantially improve the decision-making for clinical imaging studies in accordance with ACR guidelines. Specifically, a context-based algorithm performed superior to its generic counterpart, demonstrating the value of tailoring AI solutions to specific healthcare applications.
Competing Interest Statement
The authors have declared no competing interest.
Funding Statement
This study did not receive any funding
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.
Yes
Data Availability
The code for recreating the presented accGPT will be made available as an interactive interface using Gradio (https://gradio.app/) on an open Git repository as a jupyter notebook. Further code used in our analysis is available from the corresponding author upon reasonable request.
