Why test ChatGPT on the US medical licensing exam?
In the US, the USMLE is a series of three exams that doctors must pass to get a medical license. Step 1 focuses on basic science, Step 2CK on clinical reasoning, and Step 3 is taken by new doctors in training. The questions describe complex patient cases with several possible diagnoses.
ChatGPT is a large language model: a program trained on huge amounts of text that writes sentences by estimating which words are most likely to come next. It was not built for medicine. The researchers wanted to see how far it could go on such a demanding exam.
What did the researchers do?
The team took 376 questions released in June 2022 on the official USMLE website, so they were not part of ChatGPT's training data. Questions with images were removed, leaving 350.
Each question was asked in three ways: as an open question, as a multiple-choice question, and as a multiple-choice question where ChatGPT had to explain every option. A new chat was started each time. Two physicians separately graded accuracy, whether the explanation matched the answer, and whether it offered useful, non-obvious insights.
The results: close to a passing score
Leaving aside answers where ChatGPT could not decide, it scored over 50% on all three steps, and over 60% in some analyses. The passing threshold is around 60%. When undecided answers were counted as wrong, scores were lower, sometimes under 40%. Step 1, seen as the hardest, was usually its weakest.
ChatGPT's explanations matched its answer 94.6% of the time. And 88.9% of its answers included at least one insight judged useful for a student. When it got things wrong, it was mostly because of missing information and indecision.
Key figures
- 350official sample questions given to ChatGPT, with no images
- Over 50%correct answers on all three steps, and over 60% in some analyses
- 60%roughly: the USMLE passing threshold, which varies by year
- 94.6%of answers were consistent with ChatGPT's own explanation
Plain-language answers, decisions with your doctor
This study shows that an AI can answer medical exam questions well, not that it knows how to care for patients. Decisions about your health are made with your care team, who know your history.
Mima is not a doctor. She explains your symptoms, treatments, and side effects in plain words, then gathers whatever is still unclear for your appointment. One question to ask: "Does what I read about this treatment apply to my situation?"
What this means for patients
This study is about medical education, not patient care. The authors think this kind of AI could help students study, and perhaps one day support medical decisions. At their own clinic, staff tried it to rewrite radiology reports in plain language for patients.
Passing a multiple-choice test is not the same as examining a patient. A chatbot can be confidently wrong, so its answers do not replace advice from a doctor.
What are the limits of this study?
The number of questions was small, and they were practice questions, not the real exam. Questions with images, which are common in medicine, were left out. Grading by physicians is partly subjective.
The study tested the version of ChatGPT available in late 2022, and these models have changed a lot since. The authors also say studies with real students are needed to know whether ChatGPT actually helps them learn.
Frequently asked questions
Did ChatGPT pass the USMLE?
On 350 official USMLE sample questions, ChatGPT scored at or near the passing threshold of about 60%. It did not sit the real exam under official conditions.
What score do you need to pass the USMLE?
According to the study authors, the passing threshold varies by year but is about 60% of questions answered correctly.
Can ChatGPT be used for medical diagnosis?
This study does not show that. It measured knowledge on multiple-choice questions, not the ability to care for patients, and a chatbot can be wrong, so a doctor's advice is still needed.
Which version of ChatGPT was tested?
The version available in late 2022, based on the GPT-3.5 model. Newer versions have been released since.
The original study
Kung TH, Cheatham M, Medenilla A, et al. “Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models” PLOS Digital Health, February 9, 2023. DOI : 10.1371/journal.pdig.0000198
Read the full study in PLOS Digital HealthThis article summarizes a scientific study and is not medical advice.