Why does this question matter?
Large language models such as ChatGPT do well on medical exams, and many doctors are starting to use them. But passing an exam is not the same as helping a doctor reason better about a patient.
Diagnostic reasoning is the whole path to a diagnosis: considering several possible conditions, weighing the clues for and against each one, and then choosing the next tests. The researchers wanted to know whether a chatbot improves that path.
How was the trial run?
In late 2023, 50 doctors trained in family medicine, internal medicine or emergency medicine (26 senior doctors and 24 residents) took part, online or in person. They were randomly split into two groups: one could use ChatGPT Plus (GPT-4) as well as usual resources such as medical reference databases or a search engine, the other used the usual resources only.
Each doctor had 60 minutes to work through up to 6 written cases built from real patients. Experts who did not know which group each answer came from scored the reasoning with a detailed grid. The researchers also had the chatbot answer the same cases on its own.
What were the results?
Doctors with ChatGPT had a median score of 76%, compared with 74% for those using only their usual tools. This 2-point gap was not statistically significant, so it could be down to chance. Time spent per case did not change clearly either.
On its own, the chatbot scored 16 points higher than the usual-resources group. The authors conclude that having the tool is not enough: doctors need to learn how to work well with it.
Key figures
- 50doctors randomly split into two groups (with or without ChatGPT)
- 76% vs 74%median reasoning score with ChatGPT and with usual resources
- 16 pointshigher score for ChatGPT used on its own, compared with the usual-resources group
Bringing your doctor a clear account
This study is a reminder that a tool alone does not do the job: what matters is good information and how people work together. Your account of your symptoms is still a key part of your doctor's reasoning.
Mima is not a doctor and does not diagnose anything. It helps you note how you feel day by day and prepare your questions, so your doctor gets precise information. A question to ask: "What other causes should we rule out?"
What could this mean for patients?
For now, this study calls for caution rather than excitement. Giving a doctor a chatbot does not automatically improve the quality of their reasoning.
The chatbot's higher score on its own does not mean people should rely on it. The cases were written and complete, whereas in real life information arrives in pieces and the person needs to be examined. The authors call for training clinicians and designing better ways for people and AI to work together.
What are the limits?
The trial involved only 50 doctors and 6 cases, which cannot cover the variety of real situations. It tested a single tool in late 2023, while these technologies change quickly.
The doctors had no specific training in how to word their questions to the chatbot. Finally, no real patients were involved: the effect on people's health has yet to be measured.
Frequently asked questions
Does ChatGPT help doctors make a diagnosis?
In this trial, not clearly: 76% with ChatGPT against 74% without, a gap that may be due to chance.
Should I ask ChatGPT for a diagnosis instead of a doctor?
No. The chatbot alone scored better on written, complete cases, which does not reflect a real consultation with an examination and follow-up.
How many doctors took part?
50 doctors, 26 senior doctors and 24 residents, in family, internal or emergency medicine.
Which AI was tested?
ChatGPT Plus, using OpenAI's GPT-4 model, alongside the usual medical resources.
The original study
Goh E, Gallo R, Hom J, et al. “Large Language Model Influence on Diagnostic Reasoning: A Randomized Clinical Trial” JAMA Network Open, October 1, 2024. DOI : 10.1001/jamanetworkopen.2024.40969
Read the full study in JAMA Network OpenThis article summarizes a scientific study and is not medical advice.