AI matches expert severity ratings in half of hospital adverse event reports

Hospital safety systems depend on staff reading adverse event reports and judging how severe each incident was, a step that varies from reviewer to reviewer. A new single-center study tested whether a large language model could shoulder some of that burden, and the results show both promise and clear limits. Researchers at a tertiary academic medical center compared an LLM-based classification workflow against adjudicated expert judgments on 286 adverse event reports, finding exact agreement in just under half of cases but strong ordinal agreement overall.

The study, published in BMC Medical Informatics and Decision Making, was designed to avoid the common pitfall of tuning an AI system to the very data used to evaluate it. The LLM workflow’s configuration was fixed before evaluation, with no training, fine-tuning, calibration, or prompt modification on the study dataset. When the workflow generated clarification questions, the recorded responses were used to produce a second classification. Two quality and safety experts independently classified each report on an A–I severity scale aligned with the National Coordinating Council for Medication Error Reporting and Prevention framework, and a third senior reviewer adjudicated disagreements to establish the final human classification.

The human process itself revealed substantial variability. The two initial reviewers agreed exactly on 187 of 286 reports, or 65.4 percent, with moderate inter-rater reliability: an unweighted Cohen’s kappa of 0.563 and a linear weighted kappa of 0.740. Against that adjudicated human standard, the LLM’s final classification matched exactly in 140 of 286 cases, or 49.0 percent. Another 125 cases, 43.7 percent, differed by one category within a prespecified A–E near-match range, while 21 cases, 7.3 percent, showed larger discordances. The linear weighted Cohen’s kappa between the model and the final human classification was 0.728, with a bootstrap 95 percent confidence interval of 0.678 to 0.770.

Performance was notably better on the most serious reports. Among the 48 severe-harm cases in categories F through I, the model matched the final human classification exactly in 37 cases, or 77.1 percent, and mismatched in 11, or 22.9 percent. That concentration of accuracy at the severe end matters for triage, but the authors caution that the clinically important discordances were skewed toward under-classification, meaning the model sometimes rated incidents as less severe than the expert panel did. Any deployment would need safeguards against serious events being downgraded.

Efficiency was the starkest contrast. Human reviewers spent approximately 61 cumulative hours classifying the 286 reports, while the AI inference took 4 minutes and 47 seconds. The authors frame the finding as support for LLM-assisted severity classification as supervised decision support rather than autonomous adjudication, with humans retaining final judgment. They also stress the study’s limitations: it was conducted at a single center, and prospective, multicenter, and independently validated studies are needed before such systems could be relied upon in practice.

Subject of Research: Agreement between a large language model and expert human classification of adverse event severity in hospital safety reports

Article Title: Large language model–based classification of adverse event severity: agreement with final human classification in a single-center study

Article References: Gutkind, A., Shahar, L. B., Beni, S. A., Ribakovsky, I., Goldanov, Y., Abou-Arishah, R., Levy, Y., & Edden, Y. (2026). Large language model–based classification of adverse event severity: agreement with final human classification in a single-center study. BMC Medical Informatics and Decision Making. https://doi.org/10.1186/s12911-026-03924-0

Image Credits: AI Generated

DOI: 10.1186/s12911-026-03924-0

Keywords: large language models, adverse event reporting, patient safety, severity classification, NCC MERP, inter-rater reliability, clinical decision support, hospital safety monitoring, artificial intelligence, medical informatics, Large, language