Large Language Model Uncertainty Measurement and Calibration for Medical Diagnosis and Treatment

Introduction The inability of Large Language Models (LLMs) to communicate uncertainty is a significant barrier to their use in medicine. Before LLMs can be integrated into patient care, the field must assess methods to measure uncertainty in ways that are useful to physician-users.

Objective Evaluate the ability for uncertainty metrics to quantify LLM confidence when performing diagnosis and treatment selection tasks by assessing the properties of discrimination and calibration.

Methods We examined the discrimination and calibration of Confidence Elicitation, Token-Level Probabilities, and Sample Consistency metrics across GPT3.5, GPT4, Llama2-70B and Llama3-70B. Uncertainty metrics were evaluated against three datasets of open-ended patient scenarios.

Results Sample Consistency methods outperformed Token Level Probability and Confidence Elicitation methods. Sample Consistency by sentence embedding cosine similarity achieved the highest discrimination performance with poor calibration, while Sample Consistency by GPT annotation achieved the second-best discrimination with more accurate calibration. Nearly all uncertainty metrics had better discriminative performance with diagnosis questions rather than treatment selection questions and verbalized confidence (Confidence Elicitation) was found to consistently over-estimate model confidence.

Conclusions Sample Consistency methods are the optimal metrics for assessing LLM uncertainty for the tasks of medical diagnosis and treatment selection. We suggest Sample Consistency by sentence embedding cosine similarity if the user has a set of reference cases with which to re-calibrate their results, and Sample Consistency by GPT annotation if the user does not have reference cases and requires accurate raw calibration. Our results also confirm LLMs are consistently over-confident when verbalizing their confidence through Confidence Elicitation.