ChatGPT Influence on Medical Decision-Making, Bias, and Equity: A Randomized Study of Clinicians Evaluating Clinical Vignettes

In a randomized, pre-post intervention study, we evaluated the influence of a large language model (LLM) generative AI system on accuracy of physician decision-making and bias in healthcare. 50 US-licensed physicians reviewed a video clinical vignette, featuring actors representing different demographics (a White male or a Black female) with chest pain. Participants were asked to answer clinical questions around triage, risk, and treatment based on these vignettes, then asked to reconsider after receiving advice generated by ChatGPT+ (GPT4). The primary outcome was the accuracy of clinical decisions based on pre-established evidence-based guidelines. Results showed that physicians are willing to change their initial clinical impressions given AI assistance, and that this led to a significant improvement in clinical decision-making accuracy in a chest pain evaluation scenario without introducing or exacerbating existing race or gender biases. A survey of physician participants indicates that the majority expect LLM tools to play a significant role in clinical decision making.


Patient actor script:
I'm so glad you were able to see me this afternoon.Ever since I retired a few years ago at 65, I've had time to try to get healthier.I know I'm overweight, so I've started to exercise more.After my walk this morning, I noticed a weird, tired feeling that made me feel a little short of breath.I sat down in my kitchen to get a sip of water and rest; it felt better a few minutes afterwards.I also felt fine when I walked up the stairs to your office.The medical assistant who took my vital signs said everything looks great, and I've been taking the blood pressure and cholesterol medication every day.So, I don't think it's a big deal, but I want to make sure since my dad had a heart attack in his early 60's.Specifically, the patient has a HEART score of 6 (1 point for moderately suspicious history; 1 point for repolarization disturbance; 2 points for age >65; 2 points for 3+ risk factors ) without a troponin level.
Even a mild troponin increase would place the patient at 7 points (or high risk).Intermediate risk individuals were found to have a 12 -16.6 % risk of a major adverse cardiac event (MACE) within 30 days and high-risk individuals had a 50-65% risk of a MACE.Without knowledge of a troponin level, this moderate-risk individual was further evaluated and admitted to the hospital in the HEART study.
A study within primary care created a risk score to determine the likelihood of coronary artery disease in patients with chest pain.From that study, the patient has a risk score of 3 (age 65+; worse during exercise; concern about cardiac origin) and potentially 4 (if pain is not reproducible by palpation).Patients with a risk score of 3+ were found to have 4.5 times likelihood of having coronary artery disease.The recommendation for this patient who also has T-wave abnormalities is for same day troponin testing or further evaluation in the emergency department.
Given both these studies, the patient needs to be immediately evaluated for further risk stratification via cardiac enzymes or a same day noninvasive stress testing, and therefore option C is the correct answer.
Option A does not pursue necessary further evaluation.Option B delays this evaluation.Option D is not appropriate for an individual with intermediate risk.They are also on medication for blood pressure and cholesterol.Score: 2 5. Troponin (T): This data is not given.We'll assume it's 0 for the sake of this question.
Adding these together gives a HEART score of 7, which places the patient in the moderate risk category (score 4-6), corresponding to a 12 -17% chance of MACE within 6 weeks.
However, please note that this calculation is not entirely accurate as we don't have the troponin levels.A blood test for troponin levels is critical for the definitive diagnosis, and the score could be higher or lower depending on that result.It is important to consider further diagnostic testing like stress testing or angiography based on the patient's clinical situation and risk tolerance.
would be your next step in management of this patient?a. Start a daily baby aspirin and provide clear return precautions.Schedule the patient to return in one week.b.Start a daily baby aspirin and refer the patient for an urgent stress test within 2-3 days.c. Provide a full-dose aspirin and refer to the emergency department for evaluation and monitoring.d.Provide a full-dose aspirin and contact cardiology for urgent cardiac catheterization.Correct Answer: C full-dose aspirin and refer to the emergency department.Based on several risk stratification scores, this patient is at intermediate/moderate risk due to these components: -symptoms (discomfort with exertion, dyspnea) -history (concern for cardiac origin) -age (>65 years old) -EKG (T-wave inversion / flattening) -risk factors (hypertension, hyperlipidemia, obesity, positive family history) Haasenritter J, Becker A, Karatolios K, Vaucher P, Gencer B, et al.Ruling out coronary artery disease in primary care: development and validation of a simple prediction rule.Can Med Assoc J. 2010;182:1295-300.Ebell MH.Evaluation of chest pain in primary care patients.Am Fam Physician.2011;83:603-5.Poldervaart JM, Reitsma JB, Backus BE, Koffijberg H, Veldkamp RF, ten Haaf ME, et al.Effect of Using the HEART Score in Patients With Chest Pain in the Emergency Department: A Stepped-Wedge, Cluster Randomized Trial.Ann Intern Med.2017;166:689-697.Question 2: With the current information (and using risk classification tools such as the HEART score), what is this individual's chance of having a major adverse cardiac event within the next 12 -17 % within 30 days based on a HEART score of 6 (1 point for moderately suspicious history; 1 point for repolarization disturbance; 2 points for age >65; 2 points for 3+ risk factors ) without a troponin level.Reference: Mahler, S. A., Riley, R. F., Hiestand, B. C., Russell, G. B., Hoekstra, J. W., Lefebvre, C. W., ... & Herrington, D. M. (2015).The HEART pathway randomized trial.Circulation: Cardiovascular Quality and Outcomes, 8(2), 195-203.Question 3: The same patient had worsening chest pain later in the day, and on evaluation was found to have elevated troponin and ST depressions on their EKG.There were no signs of heart failure or cardiogenic shock.In addition to potential catheterization, which of the following medical interventions are most likely to improve the patient's clinical outcome?a. Aspirin, Heparin, Nitroglycerin, Supplemental oxygen b.Aspirin, Calcium antagonists (e.g., nifedipine), Morphine, Nitroglycerin, Supplemental oxygen, Statin (e.g., atorvastatin) c.ACE inhibitors (e.g., lisinopril), Aspirin, Calcium antagonists (e.g., nifedipine), Morphine, Nitroglycerin d.ACE inhibitors (e.g., lisinopril), ADP receptor inhibitors (e.g., clopidogrel), Aspirin, Beta-blockers (e.g., metoprolol), Statin (e.g., atorvastatin) Newby L, et al. 2023 AHA/ACC/ACCP/ASPC/NLA/PCNA Guideline for the Management of Patients With Chronic Coronary Disease.J Am Coll Cardiol.2023 Aug, 82 (9) 833-955.Question 4: After inpatient treatment and stabilization, the patient has been discharged and is being reviewed in a cardiology outpatient clinic several months later.Which of these medications have been shown to reduce mortality for patients with stable chronic ischemic heart disease?a. ADP receptor inhibitors (e.g., clopidogrel), Aspirin, Beta-blockers (e.g., metoprolol, carvedilol), Statins (e.g., atorvastatin, rosuvastatin), Sacubitril/Valsartan (neprilysin inhibitor) b.Anti-Arrhythmics (e.g., amiodarone), Anti-coagulants (e.g., warfarin, rivaroxaban, apixaban), Calcium Channel Blockers (e.g., amlodipine), Nitrates (e.g., isosorbide dinitrate), Oxygen supplementation c.Beta-blockers (e.g., metoprolol, carvedilol), Loop Diuretics (e.g., furosemide), Implantable Cardioverter Defibrillator (ICD), Mineralocorticoid Receptor Antagonists (e.g., spironolactone) d.ADP receptor inhibitors (e.g., clopidogrel), Angiotensin-Converting Enzyme (ACE) inhibitors / Angiotensin Receptor Blockers (ARBs) (e.g., lisinopril, losartan), Aspirin, Beta-blockers (e.g., metoprolol, carvedilol), Empagliflozin, Statins (e.g., atorvastatin, rosuvastatin) Newby L, et al. 2023 AHA/ACC/ACCP/ASPC/NLA/PCNA Guideline for the Management of Patients With Chronic Coronary Disease.J Am Coll Cardiol.2023 Aug, 82 (9) 833-955.m so glad you were able to see me this afternoon.Ever since I retired a few years ago at 65, I've had time to try to get healthier.I know I'm overweight, so I've started to exercise more.After my walk this morning, I noticed a weird, tired feeling that made me feel a little short of breath.I sat down in my kitchen to get a sip of water and rest; it felt better a few minutes afterwards.I also felt fine when I walked up the stairs to your office.The medical assistant who took my vital signs said everything looks great, and I've been taking the blood pressure and cholesterol medication every day.So, I don't think it's a big deal, but I want to make sure since my dad had a heart attack in his early 60's."ECG showed T wave inversion/flattening. Question 1: What would be your next step in management of this patient?a. Start a daily baby aspirin and provide clear return precautions.Schedule the patient to return in one week.b.Start a daily baby aspirin and refer the patient for an urgent stress test within 2-3 days.If the ECG showed T wave inversion or flattening, this could potentially indicate underlying cardiac issues such as ischemia, which would warrant further investigation.In this case, the appropriate next step would be: C. Provide a full-dose aspirin and refer to the emergency department for evaluation and monitoring.This approach allows for a more thorough assessment of the patient's condition in a timely manner.The emergency department can perform further diagnostic testing, such as cardiac enzymes and additional imaging studies, to better understand the cause of the patient's symptoms and to rule out an acute coronary syndrome or other serious cardiac conditions.Additionally, the patient can be monitored in the emergency department to ensure their safety in case their Question 2: With the current information (and using risk classification tools such as the HEART score), what is this individual's chance of having a major adverse cardiac event within the next 30 days? is a clinical tool used to estimate the risk of a major adverse cardiac event (MACE) within the next 6 weeks.It is calculated based on five factors, each scored from 0-2: 1. History (H): This patient's history is slightly suspicious.They have started to feel short of breath during exercise, which improved with rest.This could be indicative of exertional angina.Score: 1 2. ECG (E): T wave inversion or flattening is abnormal.Score: 2 3. Age (A): The patient is older than 65.Score: 2 4. Risk Factors (R): The patient is overweight and has a family history of heart disease (father had a heart attack in his early 60s).