Can an AI pass a medical exam?

Why test an AI on medical questions?

Large language models are programs trained on huge amounts of text to write answers in everyday language. Many people already ask them health questions.

In medicine, a wrong answer can cause harm. The authors note that the usual tests are often limited to multiple-choice questions marked automatically, which is not enough to judge whether an answer is safe and useful.

How was the study done?

The Google team built MultiMedQA, a collection of seven question sets: medical exam questions, research questions and questions from the public. One of them, HealthSearchQA, holds 3,173 health questions that people commonly type into search engines.

They tested PaLM, a model with 540 billion parameters (its internal settings), and Flan-PaLM, a version trained to follow instructions. Then they built Med-PaLM, a version adapted to medicine using a few examples written by doctors.

Beyond quizzes, doctors reviewed the longer answers against several criteria: accuracy, understanding, reasoning, possible harm and bias.

What did the study find?

On US medical licensing exam-style questions, Flan-PaLM scored 67.6%, more than 17 points above the best result published until then.

But the doctors' review showed gaps. Only 61.9% of Flan-PaLM's longer answers were judged in line with scientific consensus, and 29.7% could lead to harm.

With Med-PaLM, 92.6% of answers were in line with consensus, on a par with answers written by doctors (92.9%). Potentially harmful answers fell to 5.9%, close to doctors (5.7%). Even so, the authors conclude that Med-PaLM remains inferior to clinicians.

What does this mean for patients?

This study puts no tool in patients' hands. It shows that these models hold a lot of medical knowledge, but that a good exam score does not guarantee a safe answer.

Above all, it offers a way to evaluate these tools with doctors, beyond quizzes. For patients, the message is clear: an AI answer can help you understand something or prepare a question, not replace a clinician's advice.

What are the limits of this study?

The questions are tests, not real patients with their history, results and treatments. Passing a quiz says nothing about the ability to care for someone.

The authors describe the doctors' review as a pilot evaluation, on a limited number of answers and reviewers. They call for further evaluation, especially on safety, fairness and bias. And these models change fast: the figures describe the 2023 versions.