Why is summarizing medical records such a burden?
Doctors spend a lot of time summarizing text: imaging reports, progress notes, treatment histories. The authors cite data showing that physicians can spend up to 2 hours on documentation for every hour with patients.
This work is tiring and can lead to mistakes. Large language models, AI systems like ChatGPT that read and write text, could help. But first, someone had to check whether they summarize as well as professionals without adding errors.
How did the Stanford team run the study?
The researchers tested 8 language models on 4 tasks: summarizing radiology reports, patient questions, hospital progress notes, and doctor-patient conversations. They adapted the models in two ways: by adding examples to the instructions, or by lightly retraining them. The best results came from GPT-4 guided by examples.
Next, 10 physicians (5 radiologists and 5 hospitalists, doctors who care for patients in the hospital) compared its summaries with those written by human experts, on 100 documents per task, for 3 of the 4 tasks. They did not know which summary came from the AI. They rated completeness (nothing important left out), correctness, and conciseness.
What did the study find?
The AI summary was preferred in 36% of cases and rated equivalent in 45%. The expert summary was preferred in only 19%. On average, the AI summaries were more complete, without being longer, and contained fewer errors.
The team also counted made-up information, known as hallucinations. The AI produced them in 5% of summaries, compared with 12% for the human experts. It was not flawless: in one example, it listed health problems that were not in the record. In a harm analysis, physicians judged that the expert summary was slightly more likely to cause harm (14%) than the AI summary (12%).
Key figures
- 36%of cases where physicians preferred the AI summary
- 45%of cases where both summaries were rated equivalent
- 19%of cases where the human expert's summary was preferred
- 5% vs 12%of summaries with made-up information: AI vs. experts
Keeping your own health story with Mima
These summary tools are aimed at doctors first, to lighten their paperwork. In the study, they were tested on existing documents, not during visits, and a professional still has to check what they write.
On your side, Mima keeps track of your appointments, prescriptions, and questions, and turns what you tell her into a clear picture of your symptoms over time. Before a visit, she prepares your questions, such as: "Can we go over the summary in my record together?"
What could this mean for patients?
If tools like this are built into hospital record systems, they could offer a first draft summary that a doctor reviews and corrects. The authors say this could ease the documentation burden and free up more time for patient care.
For patients, the benefit would be indirect: doctors spending less time on paperwork, and more complete record summaries. That still has to be shown in real hospitals.
What are the limits of this study?
The study used existing documents from public datasets, not live patient care. The authors note that some of these texts may have been part of GPT-4's training data, which could give it an edge.
The AI could not see the patient's earlier exams. Different specialists may also want different kinds of summaries, which the study did not test. The authors call for prospective studies in real clinical settings and flag the risk of bias as something to watch.
Frequently asked questions
Can AI summarize medical records accurately?
In this study, GPT-4 guided by examples wrote summaries that physicians rated as equivalent or better than expert summaries in most cases. It has not yet been tested in real clinical workflows.
Does AI make up information in medical summaries?
Sometimes. Made-up information appeared in 5% of AI summaries, compared with 12% of human expert summaries. A clinician still needs to review the output.
What kinds of medical documents were summarized?
Radiology reports, patient questions, hospital progress notes, and doctor-patient conversations.
Will AI replace doctors' notes?
No. The idea is for the AI to draft a summary that the doctor checks and approves, saving time on documentation.
The original study
Van Veen D, Van Uden C, Blankemeier L, et al. “Adapted large language models can outperform medical experts in clinical text summarization.” Nature Medicine, February 27, 2024. DOI : 10.1038/s41591-024-02855-5
Read the full study in Nature MedicineThis article summarizes a scientific study and is not medical advice.