Why test ChatGPT on hard diagnostic cases?
Generative AI models like GPT-4 answer written questions with detailed text. They score well on medical exams. But passing a multiple-choice test is not the same as figuring out what is wrong with a patient when the signs are rare or misleading.
To measure this, the researchers used a long-standing tool: the clinicopathological conferences of the New England Journal of Medicine, one of the most widely read medical journals. Each one tells the story of a real, complicated case, then reveals the final diagnosis, confirmed by examining tissue under a microscope. Since the 1950s, these cases have been used to test diagnostic software.
How did the researchers run the test?
Three physicians at Beth Israel Deaconess Medical Center in Boston took the cases published from January 2021 to December 2022. For each one, they copied the case story up to the point where the expert starts reasoning, then pasted it into GPT-4 with the same instruction: list the possible diagnoses, from most to least likely.
Each case ran in a separate chat so the model could not learn from earlier ones. Of 80 cases, 10 were left out (not a true diagnostic puzzle, or too long). Two doctors scored every answer on a 0 to 5 scale, and a third settled disagreements.
Results: often on the list, less often at the top
Across the 70 cases, the correct diagnosis was on GPT-4's list 64% of the time (45 cases). It was ranked first 39% of the time (27 cases). The list held 9 diagnoses on average, and when the right one was there, it sat around second or third place on average.
The average quality score was 4.2 out of 5. The authors say this compares well with older diagnostic software: an earlier version of one such program scored 3.8 on the same scale.
Key figures
- 70difficult NEJM case challenges analyzed (out of 80, 10 excluded)
- 64%of cases where the correct diagnosis appeared in GPT-4's list of possibilities
- 39%of cases where GPT-4's top diagnosis was the right one
- 9possible diagnoses listed by GPT-4 on average for each case
Walking into your appointment with the right questions
This study does not change your care: GPT-4 was tested on published cases, and diagnosis stays with your doctor, who examines you and orders the right tests.
Mima does not diagnose anything. She helps you describe your symptoms day by day as a clear picture, and gathers your questions before the visit, like: "What other possibilities did you consider?" She also explains medical terms in plain words.
What this does, and does not, mean for patients
The study suggests generative AI could help doctors think through hard cases by reminding them of possibilities they might miss. The authors call it a "promising adjunct" to human reasoning.
It does not show that GPT-4 can diagnose in place of a doctor. The model received complete case write-ups prepared by experts, while in a real visit someone has to ask questions, examine the patient, and order tests. If a symptom worries you, a health care professional is still the one to see.
What are the limits of this study?
The sample is small: 70 cases from a single journal. These cases are puzzles picked for their difficulty, not everyday primary care. GPT-4 was not compared with physicians on the same cases.
Scoring is partly subjective: the two main scorers agreed on only 66% of scores. The model is also a "black box" whose biases and blind spots are unknown. The authors call for studies using real patient visits once privacy concerns are addressed.
Frequently asked questions
Can ChatGPT diagnose medical conditions?
In this study, GPT-4 listed the correct diagnosis for 64% of 70 published cases. That does not make it a doctor: it cannot examine you, and the study did not involve real patients.
What are NEJM clinicopathological cases?
They are real, difficult cases published by the New England Journal of Medicine, with the final diagnosis revealed at the end. They have long been used to test diagnostic tools.
Is GPT-4 better than doctors at diagnosis?
This study cannot say: GPT-4 was not compared with physicians on the same cases.
Is it safe to use ChatGPT to check my symptoms?
It can help you put questions into words, but it can be wrong. If a symptom worries you, see a doctor, and call 911 in an emergency.
The original study
Kanjee Z, Crowe B, Rodman A “Accuracy of a Generative Artificial Intelligence Model in a Complex Diagnostic Challenge” JAMA, June 15, 2023. DOI : 10.1001/jama.2023.8288
Read the full study in JAMAThis article summarizes a scientific study and is not medical advice.