Multidimensional Evaluation of Large Language Models on the AAP In-Service Examination: Assessing Accuracy, Calibration, and Citation Reliability - Dhaimade, Prita Abhay; Henderson, Robin (medRxiv, 2025)
Background Large language models (LLMs) have demonstrated rapid advancements in natural language understanding and generation, prompting their integration into biomedical research, clinical practice, and professional education. However, systematic evaluation of LLMs in specialty-specific domains such as dentistry and periodontology remain limited, particularly regarding multidimensional performance metrics.Objective To conduct a comprehensive, multidimensional assessment of commercially available LLMs: GPT-4.0, GPT-5.0, and Claude SONNET 4.0 on the American Academy of Periodontology in-service examination, focusing on response accuracy, self-assessed confidence calibration, citation validity, and hallucination prevalence.Methods Models were evaluated on the 2024 AAP In-Service Examination (331 questions) using two formats: Full Test (all questions at once) and Individual Question (one at a time). Prompts were standardized; models selected answers, and for GPT-5.0 and Claude SONNET 4.0, also provided confidence ratings and citations. Citation validity was assessed using a human-in-the-loop protocol with expert review. Statistical analyses included chi-square, McNemar’s, and logistic regression to assess accuracy, question fatigue, confidence calibration, and citation reliability.Results LLMs achieved high overall accuracy (78–87%), with the Individual Question format consistently yielding higher scores than Full Test, though differences were not statistically significant.Accuracy was highest in fact-dense domains (biochemistry, physiology, microbiology) and lowest in integrative domains (diagnosis, therapy). Significant question fatigue was observed in GPT-5.0 Full Test mode (OR = 0.997, p = 0.035), but not in Individual Question mode.Confidence scores predicted accuracy, with the strongest calibration in Individual Question mode. Citation analysis revealed frequent hallucinations, mostly critically erroneous, and citation validity was independent of answer accuracy.Conclusions LLMs can answer a broad spectrum of periodontal specialty questions, but their reliability varies with context and information presentation. While promising as adjunctive tools, their outputs— especially for complex reasoning and citations—require rigorous human review in educational and research settings to ensure accuracy and safety.Author Summary Artificial intelligence chatbots are rapidly entering medical education, yet we lack comprehensive understanding of their reliability when students depend on them for learning. We developed a multidimensional evaluation framework to systematically assess AI performance beyond simple accuracy, examining how these systems behave across different medical topics, question types, and presentation formats.Using 331 real dental examination questions, we tested three major AI systems, analyzing not only correctness but also confidence calibration - whether AI confidence levels match actual accuracy - and implementing human-in-the-loop verification to check if cited sources actually exist.Our findings highlight critical vulnerabilities in current AI systems. Most alarmingly, these chatbots fabricated nearly half of their citations while maintaining unwavering confidence in both correct and incorrect responses. This combination of overconfidence and misinformation means students cannot distinguish reliable from unreliable AI responses. Additionally, we documented progressive performance decline during sequential questioning, similar to human cognitive fatigue.While we know AI systems generate rather than retrieve information, our research demonstrates the real-world consequences of this limitation. As artificial intelligence integrates into education, healthcare diagnostics, and insurance decisions, these findings underscore the urgent need for better evaluation frameworks and user education about AI limitations.Competing Interest StatementThe authors have declared no competing interest.Funding StatementThe author(s) received no specific funding for this work.Author DeclarationsI confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.Not ApplicableThe details of the IRB/oversight body that provided approval or exemption for the research described are given below:This research does not involve human subjects. The study evaluated the performance of artificial intelligence language models (GPT-4o, GPT-4o mini, and Claude-3.5 Sonnet) on publicly available American Academy of Periodontology examination questions. No human participants were recruited, interviewed, surveyed, or involved in any aspect of data collection. All analyses were conducted on AI-generated text responses and publicly available examination materials. Therefore, Institutional Review Board (IRB) review was not required for this computational research study.I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.Not ApplicableI understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).Not ApplicableI have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.Not ApplicableThe datasets generated and analyzed during this study are available from the corresponding author upon reasonable request. The complete dataset includes: AI model responses to all 331 American Academy of Periodontology examination questions, including accuracy assessments, confidence scores, and generated citations for GPT-4o, GPT-4o mini, and Claude-3.5 Sonnet models Human expert validation results for all AI-generated citations, including verification status and authenticity scores Statistical analysis outputs including confidence calibration metrics, accuracy measurements across question categories, and question fatigue analysis. Data requests should be directed to the corresponding author