Single-run benchmark finds AI liver-injury extraction errors

Large language models generated machine-readable summaries of liver injury case narratives in a source-grounded benchmark, but valid formatting did not guarantee agreement with the underlying evidence. Under the study’s single-run conditions, GPT, Codex and a heterogeneous Gemini mixed-endpoint condition produced 360 parseable outputs while exhibiting different levels of schema adherence, agreement with a curated reference and targeted safety-relevant errors. Blinded human adjudication confirmed 10, 19 and 38 potentially causality-distorting errors in the three conditions, respectively; unsupported claims of absent co-exposures were the most frequent confirmed category.

The descriptive benchmark used 120 LiverTox-derived narratives held out from prompt and schema development. The population comprised 42 herb-induced or botanical/herbal dietary supplement-associated liver injury cases and 78 conventional drug-induced liver injury comparators. A locked schema defined 40 fields for each case, addressing information such as exposure chronology, dechallenge, rechallenge, co-exposures, competing causes, injury phenotype, product characteristics and outcome. This was not an evaluation of clinical decision-making or automated causality assessment: the researchers tested whether three execution conditions could produce structured extractions that were traceable to, and consistent with, the source narratives.

All 360 outputs across the three conditions were parseable. Valid-status rates were 100.0% for GPT, 99.9% for Codex and 63.5% for the heterogeneous Gemini mixed-endpoint condition. Agreement with the final curated reference was 81.5% for both GPT and Codex, compared with 50.3% for Gemini. Normalized exact-match rates were 60.0%, 60.5% and 30.8%, respectively. The reference used for comparison was developed through an initial single-expert assessment followed by a model-triggered, model-blinded human source re-audit of 240 unique case-field rows. That audit yielded 184 corrections before the reference was rescored, illustrating that agreement depended on a carefully curated comparison standard rather than an unchanged first annotation.

The models also performed well on a narrower measure of textual fidelity. Among nonblank evidence spans they supplied, verbatim match rates were 100.0% for GPT, 99.8% for Codex and 95.6% for Gemini. However, faithful reproduction of selected passages did not eliminate potentially consequential discrepancies elsewhere in the structured extraction. A six-rule targeted safety-discrepancy screen generated 15 flags for GPT, 25 for Codex and 40 for Gemini. After blinded human adjudication, 10, 19 and 38 flags, respectively, were confirmed as potentially causality-distorting model errors, with unsupported absence of co-exposures the most frequent confirmed category. The findings therefore distinguish syntactic validity and quotation accuracy from the broader reliability of extracted evidence.

The results are limited to a descriptive, single-run benchmark involving 120 derived narratives, one locked 40-field schema and three evaluated execution conditions. The study did not evaluate automated causality assessment, clinical readiness or the effectiveness of a human-in-the-loop workflow, so it cannot establish whether model-assisted review would improve pharmacovigilance outcomes in practice. It also does not show that the targeted error categories were broadly representative beyond the narratives and procedures tested. Within that scope, however, the benchmark demonstrates that parseable outputs and near-perfect verbatim matches for provided evidence spans are not sufficient proof that an AI system has accurately extracted the full set of causality-relevant information from a case.

Subject of Research: Single-run benchmark of AI extraction of causality-relevant evidence from 120 liver injury case narratives

Article Title: A source-grounded benchmark for extracting causality-relevant pharmacovigilance evidence from liver injury case narratives

Article References: Kim, T.-Y., & Kim, J.-H. (2026). A source-grounded benchmark for extracting causality-relevant pharmacovigilance evidence from liver injury case narratives. BMC Pharmacology and Toxicology. https://doi.org/10.1186/s40360-026-01247-4

Image Credits: AI Generated

DOI: 10.1186/s40360-026-01247-4

Keywords: large language models, liver injury, pharmacovigilance, case narratives, evidence extraction, benchmarking, human verification, AI safety, co-exposures, causality evidence, source-grounded, benchmark