Why does catching sepsis early matter?
Sepsis is a severe reaction of the body to an infection that can damage organs. When it is caught and treated early, especially with antibiotics, it is linked to fewer hospital deaths. In the US, several hundred thousand patients are hospitalized for sepsis every year.
To help care teams, electronic health record companies added automatic alerts. The Epic Sepsis Model, from Epic Systems, is one of the most widely used: it is a statistical model that calculates a risk score from data recorded during a hospital stay. Because it is proprietary (its inner workings are not public), no independent evaluation had been published, according to the authors.
What did the researchers do?
The team looked back at data from 27,697 adults admitted to Michigan Medicine, the University of Michigan's health system, between December 2018 and October 2019, for a total of 38,455 hospital stays. The Epic score was calculated every 15 minutes, starting in the emergency room.
The researchers compared these scores with what actually happened: who developed sepsis, and when. They used an alert threshold of 6, chosen by their hospital and within the range Epic recommends. Stays where the alert was actually shown to clinicians were left out so they would not skew the results.
The results: many alerts, many missed cases
Sepsis occurred in 2,552 hospital stays, or 7%. The tool scored 0.63 on the area under the curve, a measure that runs from 0.5 (chance) to 1 (a perfect test). That is well below the 0.76 to 0.83 reported in Epic's internal documents.
At a threshold of 6, the tool missed 1,709 patients with sepsis (67%), while flagging 18% of all hospital stays. Clinicians would have had to check 8 flagged patients to find one future sepsis case. And the tool caught early only 183 patients (7% of sepsis cases) who had not already received timely antibiotics from their care team.
Key figures
- 0.63area under the curve at the hospital-stay level (0.5 = chance, 1 = perfect)
- 67%of patients with sepsis were not flagged by the tool (1,709 of 2,552)
- 18%of all hospital stays crossed the alert threshold
- 8flagged patients to check to find a single future sepsis case
A loved one in the hospital: keeping track with Mima
These alerts run behind the scenes in the hospital's software, so patients and families never see them. The study is a reminder that even a widely used tool needs checking, and that the care team's attention still matters most.
During a hospital stay, Mima helps you keep track of treatments, conversations with the care team, and the questions that come up. For example, you could prepare this one: "Does this unit use a computer alert for sepsis, and who checks it?"
What this means for patients
A computer alert is not a diagnosis. When it goes off too often for nothing, clinicians may start paying less attention, which is called alert fatigue. The study also shows that care teams were already catching many cases: among patients the tool missed, 1,030 still received antibiotics on time.
This study does not say that all sepsis alerts are useless. It shows that a widely used tool needs to be checked in each hospital, and the authors ask medical societies to issue guidance on these algorithms.
What are the limits of this study?
The evaluation took place in a single academic hospital, although the patient group was large and diverse. The sepsis definition combined two official sets of criteria, and definitions are still debated.
How you measure matters a lot: when scores calculated up to 3 hours after sepsis began were included, the area under the curve rose to 0.80. Finally, the study used past data, so it does not measure how the alert affects patients' health in real-world use.
Frequently asked questions
What is sepsis?
Sepsis is the body's extreme, out-of-control response to an infection, which can damage organs. It is a medical emergency, treated in part with antibiotics.
What is the Epic Sepsis Model?
It is an alert tool built into Epic's electronic health record and used in hundreds of US hospitals. It calculates a sepsis risk score every 15 minutes.
How accurate is the Epic sepsis alert?
In this University of Michigan study, it scored 0.63 on the area under the curve and missed 67% of sepsis cases. Results can differ from one hospital to another, which is why local checks matter.
What is alert fatigue in hospitals?
Alert fatigue happens when clinicians get so many alerts, often false ones, that they start to tune them out. It can lead to missing the alerts that matter.
The original study
Wong A, Otles E, Donnelly JP, et al. “External Validation of a Widely Implemented Proprietary Sepsis Prediction Model in Hospitalized Patients” JAMA Internal Medicine, August 1, 2021. DOI : 10.1001/jamainternmed.2021.2626
Read the full study in JAMA Internal MedicineThis article summarizes a scientific study and is not medical advice.