Abstract
Background General purpose language models such as ChatGPT are increasingly used by physicians and triage nurses during emergency triage. A recent study reported 51.6% undertriage of emergencies when patients queried ChatGPT directly (Ramaswamy et al., 2026). DR. INFO is an agentic AI based clinical assistant that retrieves over a curated clinical knowledge base, and an MTS specific retrieval configuration is available in which the system also retrieves the Manchester Triage System (MTS) textbook at inference time. We evaluate all three configurations against the Manchester Triage System.
Methods We adapted the clinical scenarios published by Ramaswamy et al. and mapped them to the Manchester Triage System, yielding 39 emergency cases covering all five MTS priority levels. Each case was evaluated in two variants, one without and one with the objective clinical data block (vital signs, examination findings, and laboratory results), and permuted across two genders, giving 156 prompts per system per condition (4 permutations of each of the 39 cases). Three systems were tested with and without a misleading GP referral statement prepended as an anchoring statement, giving 312 prompts per system: DR. INFO Baseline, DR. INFO with MTS retrieval, and OpenAI GPT-5.1. The primary outcome was the undertriage rate on the ordered MTS scale, tested with Fisher’s exact test.
Results GPT-5.1 undertriaged 44.2% of prompts (69/156; 95% CI 36.7 to 52.1), including 75.0% of Red-priority and 73.4% of Orange-priority prompts. Both DR. INFO configurations undertriaged 11.5% of prompts (18/156; 95% CI 7.4 to 17.5; Fisher’s exact p = 1.0 × 10−10 versus GPT-5.1). The 156 prompts derive from 39 clinical cases (2 objective-data variants × 2 genders each); the 8 Red-priority prompts derive from 2 Red cases. When the anchoring statement was prepended, GPT-5.1 undertriaged 8 of 8 Red-priority prompts, while both DR. INFO configurations continued to undertriage none. Adding objective clinical data to the input reduced undertriage in DR. INFO with MTS retrieval from 19.2% to 3.8% (p = 0.005). DR. INFO Baseline and GPT-5.1 showed no comparable change. There was no statistically detectable gender difference.
Conclusion On this benchmark, replacing a general purpose language model with an agentic retrieval augmented system over a curated clinical knowledge base substantially reduced the undertriage rate. Adding retrieval of the Manchester Triage System textbook to the agentic system was further associated with an appropriate change in the assigned MTS priority when objective clinical data became available. Of the three configurations evaluated here, DR. INFO with MTS retrieval combined a clinically conservative assignment at Red priority with a reduction in undertriage when objective clinical data were added (19.2% to 3.8%).
Competing Interest Statement
The authors declare the following competing interests. Sandhanakrishnan Ravichandran, Miguel Romano, Rogerio Corga da Silva, Shivesh Kumar, Michiel van der Heijden, and Valentine Emmanuel Gnanapragasam were employed by Synduct GmbH during the drafting of this work. Tiago Mendes and Marta Isidoro served as clinical advisors to Synduct GmbH during the study. Nader Absi reports no competing interests.
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.
Yes
Footnotes
For some reason the authors list not matching the pdf. So i am resubmitting the pdf.
Data Availability
MTS-Bench is available from the corresponding author on reasonable request. The clinical scenarios from which MTS-Bench was adapted are published by Ramaswamy et al. (2026, Nature Medicine, doi:10.1038/s41591-026-04297-7). The Manchester Triage Group reference text used as the gold standard is published by Mackway-Jones et al. (2014, John Wiley & Sons/BMJ Books, ISBN 978-1-118-29906-7).
Funder Information Declared
