AMIE: Google's AI tested against primary care doctors in simulated consultations

Why test an AI in a consultation?

Conversation between doctor and patient sits at the heart of medicine. By asking the right questions, a doctor pieces together the story of the symptoms, which helps them reach a diagnosis.

Large language models, the AI systems that write text, can answer medical questions. But running a whole consultation, with listening, questions and explanations, is much harder. The researchers wanted to see how far an AI could go in this task.

How was the study run?

The team built AMIE, an AI trained partly by holding simulated consultations with itself. They then set up an exam modeled on the practical tests used for medical students, known as OSCEs.

Trained actors played patients based on 159 scenarios from Canada, the United Kingdom and India. Each actor chatted by text without knowing whether they were talking to AMIE or to one of 20 primary care doctors. Specialist doctors, also blinded, then scored every consultation for history-taking, diagnostic accuracy, suggested management, communication and empathy.

What were the results?

AMIE produced lists of possible diagnoses that were more accurate than the doctors' lists. Specialists rated it higher on 30 of 32 criteria, and the patient actors rated it higher on 25 of 26 criteria, including perceived empathy.

Both groups of raters found AMIE better at communication and empathy, which made up most of the criteria assessed. In other words, its written replies came across as more attentive and clearer than the doctors' replies in this format.

These figures are striking, but they come from a very particular setting, which the authors themselves describe as a major limitation.

What could this mean for patients?

Today, nothing changes in your care: AMIE is not a tool you can consult. The study mainly shows that AI might one day help prepare or structure a consultation, under the supervision of clinicians.

A diagnosis also relies on a physical examination, tests and knowing the person over time. None of this was part of the study.

What are the limits?

The doctors had to consult by text chat, which is unfamiliar in everyday practice and may have put them at a disadvantage. The patients were actors rather than people who were unwell, and the scenarios do not cover the full range of real situations.

The authors conclude that more research is needed before AMIE could be used in real-world settings, including work on safety, fairness and privacy.