Why are doctors swamped by patient messages?
As care moves online, patients write to their clinicians more and more often. Every message takes time to answer, on top of everything else. The authors point out that this workload feeds burnout among health professionals.
Hence an idea: an AI assistant could write a first draft of the reply, which the doctor would then check and edit. But first we need to know whether those replies are accurate and humane.
How did the researchers compare the chatbot with doctors?
The team used a public forum, r/AskDocs on Reddit, where people ask health questions and doctors with verified credentials reply. They drew 195 exchanges from October 2022 at random.
Each question was then put, word for word, to ChatGPT, a chatbot, meaning a program that answers in everyday language. A fresh session was opened each time so earlier questions could not shape the answer.
A team of licensed health professionals read each question with both answers, without knowing which came from the doctor. Every exchange was rated three times. Reviewers picked the better answer and scored the quality of the information and the empathy, meaning how well the reply put itself in the person's shoes.
What did the study find?
Across 585 evaluations, the chatbot's answer was preferred 78.6% of the time. Doctors' answers were much shorter: an average of 52 words, against 211 for the chatbot.
The share of answers rated good or very good was 78.5% for the chatbot and 22.1% for doctors. For empathy, 45.1% of the chatbot's answers were rated empathetic or very empathetic, against 4.6% of the doctors'.
The authors conclude that, in this forum setting, the chatbot produced good quality, empathetic replies.
Key figures
- 78.6%of evaluations preferred the chatbot's answer over the doctor's
- 52 vs 211 wordsaverage length of doctors' and chatbot answers
- 78.5% vs 22.1%of answers rated good or very good quality, chatbot vs doctors
- 45.1% vs 4.6%of answers rated empathetic or very empathetic, chatbot vs doctors
Getting your questions ready, with your doctor at the center
This study compares written answers on a forum, not consultations. A chatbot can sound reassuring, but it knows neither your records nor your examination. Mima is not a doctor and does not make diagnoses.
What Mima does: help you note your symptoms day by day and gather your questions before the appointment, so your doctor has time to answer them. A useful question to ask: "How can I contact you between appointments, and how quickly do you usually reply?"
What could this mean for patients?
The authors suggest testing these tools in real clinics, for example to draft a reply that the doctor reviews and edits before sending. The doctor would stay responsible for the answer.
If that worked, patients could get fuller replies and clinicians could save time. But it still has to be shown: the authors call for randomized trials to find out whether such assistants really improve replies, reduce burnout and help patients' health.
What are the limits of this study?
The questions came from a public forum, not a clinic. A doctor answering a stranger online in their spare time does not write the way they would to their own patient, whose records they know.
Reviewers judged quality and empathy, but the study did not check whether the advice actually helped anyone. The chatbot's longer replies may have swayed their preference. And a chatbot can be wrong while sounding sure of itself: it does not replace an examination, a medical record or a clinician's judgment.
Frequently asked questions
Does ChatGPT answer better than doctors?
In this study, on questions posted to an online forum, health professionals preferred the chatbot's answers in 78.6% of evaluations. It says nothing about a real appointment with a doctor who knows your records.
Can a chatbot replace my doctor?
No. The authors suggest using the chatbot to draft a reply that the doctor checks and corrects. Diagnosis and care decisions stay with clinicians.
Why did the chatbot seem more empathetic?
Its answers were longer and more detailed. The doctors were volunteering on a forum, often replying in a few sentences.
Does this study prove a health benefit?
No. It compares written answers. The authors call for randomized trials to measure the real effect on patients and clinicians.
The original study
Ayers JW, Poliak A, Dredze M, et al. “Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media Forum” JAMA Internal Medicine, June 2023. DOI : 10.1001/jamainternmed.2023.1838
Read the full study in JAMA Internal MedicineThis article summarizes a scientific study and is not medical advice.