An artificial intelligence system built from multiple cooperating language-model agents, checked by human reviewers at every step, detected immune-related adverse events in oncology notes with high accuracy in retrospective testing, held up reasonably well during three months of silent real-world deployment, and cut annotation time nearly in half for clinical research staff in a randomized crossover trial.
Immune checkpoint inhibitors have transformed treatment for many cancers, but they can trigger immune-related adverse events, or irAEs, which the study’s authors say affect up to 40% of patients receiving these drugs. Identifying these events currently depends on laborious and inconsistent manual chart review, a bottleneck for safety monitoring, clinical trial curation and real-world evidence generation. The research team, led by Jack Gallifant and Danielle Bitterman of Brigham and Women’s Hospital and Dana-Farber Cancer Institute, set out to test whether an agentic system could extract not just the presence of an irAE but also its timing, severity grade, attribution to immunotherapy and diagnostic certainty, directly from free-text clinical notes.
The architecture decomposes the task into specialized agents. A preprocessing agent formats the relevant text, then parallel pipelines for six irAE subtypes, pneumonitis, hepatitis, colitis, myocarditis, thyroiditis and dermatitis, pass each note through four sequential stages: binary detection of the event, severity grading against CTCAE v5.0 criteria, assessment of whether the clinician attributed the event to checkpoint inhibitor therapy, and estimation of diagnostic certainty. Crucially, the detection and grading stages each ran three times independently, with a judge agent reconciling disagreements, a self-consistency mechanism that proved central to performance. The system also extracts the exact text spans supporting each determination, a feature that later became the backbone of the human collaboration interface.
In retrospective testing on 263 expert-annotated clinical notes drawn from patients treated with checkpoint inhibitors at an academic medical center in Boston between 2015 and 2024, the best configuration, GPT-4.1-mini with the default agentic architecture, achieved a macro-averaged F1 score of 0.92 for detecting currently active events and 0.91 for resolved past events, with a temporality-agnostic score of 0.93 across the six conditions. Detection was strongest for hepatitis and pneumonitis (F1 = 0.96) and weakest for myocarditis (F1 = 0.86). Multi-class severity grading was harder: the macro F1 for exact CTCAE grade assignment on current events was 0.66, ranging from 0.73 for dermatitis down to 0.51 for colitis, with intermediate grades 2 and 3 accounting for most misclassifications. Attribution extraction was strong, at F1 = 0.92 for current and 0.94 for past events, and certainty classification reached 0.93 to 0.94. The best configuration cost about US$0.021 per note.
The self-consistency mechanism mattered enormously. Removing it dropped detection F1 from 0.92 to 0.78 while only halving the inference cost, confirming that multi-run consensus with judge adjudication contributed roughly 0.14 to the macro F1. Smaller proprietary models and open-source alternatives consistently underperformed. Interestingly, a simple rule-based baseline using regular expressions built from 779 terms performed comparably to or better than some language-model approaches for binary detection, but the authors note that keyword matching cannot extract severity grade, attribution, certainty or supporting evidence spans, the granular attributes needed for actionable adverse-event characterization.
Performance attenuated, as expected, when the fixed system was deployed silently in the live electronic health record for three months from May to July 2025, processing 884 notes in near-real time and generating 31,824 label adjudications. Binary F1 across grade, attribution and certainty labels was 0.72 for current events and 0.79 for past events, with attribution (F1 = 0.77) and certainty (F1 = 0.80) holding up better than grading. At the note level, model predictions matched human review in 62% of cases for grade, 87% for attribution and 83% for certainty. The authors attribute the decline to temporal drift in documentation patterns and case mix, different EHR templates, and broader clinical variation than the curated development corpus captured, rather than fundamental model limitations, and they emphasize that deployed clinical AI requires continuous monitoring and periodic recalibration, not one-time validation.
The most striking human results came from a randomized 2 × 2 crossover study in which 17 clinical research staff, 16 clinical research coordinators and one clinical research nurse, curated irAE labels from 20 notes each, alternating between standard manual review and AI-assisted annotation. AI assistance reduced median annotation time from 428 seconds to 242 seconds per note, a 43.5% reduction; the primary generalized estimating equations analysis estimated a difference of −2.71 minutes (95% confidence interval −3.89 to −1.53; P = 6.98 × 10⁻⁶). Complete-match accuracy, a stringent metric requiring all six current grade labels to exactly match the gold standard, rose from 19.4% to 25.9%, corresponding to a 45% increase in the odds of complete accuracy (odds ratio 1.45; 95% CI 1.01 to 2.08; P = 0.045).
Even more dramatic was the effect on consistency. For current grade labels, Krippendorff’s alpha, a measure of interannotator agreement, rose from 0.22 to 0.85 in one note set and from 0.51 to 0.82 in the other, with concordant improvements for attribution and certainty labels. Manual annotation produced only fair agreement among reviewers, whereas AI-assisted annotation achieved near-excellent reliability. The authors suggest annotators anchor on the model’s prepopulated suggestions, reducing idiosyncratic interpretation of ambiguous documentation, a standardization benefit they argue may be as valuable as the time savings for pharmacovigilance and registry curation. They also flag the flip side: the study was not designed to distinguish genuine consistency gains from automation complacency, and future work should test whether anchoring on AI suggestions compromises reviewers’ ability to catch model errors. Participants noticed the difference, scoring the AI-assisted workflow 67.4 versus 45.7 on the System Usability Scale, and 15 of 17 respondents preferred it.
The authors are candid about limitations. All phases, including the prospective deployment and the user study, took place within a single healthcare system with local note templates and vernacular, so no phase establishes external validation, which they call the most important open question. The system covers only six irAE subtypes, a subset of the broader CTCAE ontology, and relied on a single proprietary model accessed through a commercial API, raising cost and infrastructure concerns for smaller institutions; open-source alternatives run behind institutional firewalls eliminated API costs but performed worse with current prompting strategies. Ground-truth labels remain constrained by documentation quality and the inherent subjectivity of grading, and the prospective phase did not assess patient outcomes such as time to steroid initiation or hospitalization.
Still, the study offers a template for closing what the authors call the deployment gap in clinical AI: staging validation from retrospective benchmarking through silent prospective monitoring to embedded human collaboration, with auditable, evidence-linked outputs at each step. By surfacing candidate adverse events with attributed causality and provisional grade shortly after documentation, such systems could shorten time to recognition for life-threatening irAEs like myocarditis and pneumonitis and standardize ascertainment for research registries and safety reporting, provided external validation and outcome-focused trials confirm what this single-system study suggests.
Subject of Research: Validation of an agentic large language model system for extracting immune-related adverse event details from oncology clinical notes
Article Title: An agentic AI system enhances clinical detection of immune-related adverse events in a multi-phase validation study
Article References: Gallifant, J., Chen, S., Shin, K.-Y., Kellogg, K. C., Doyle, P. F., Guo, J., Ye, B., Warrington, A., Zhai, B. K., Hadfield, M. J., Gusev, A., Ricciuti, B., Christiani, D. C., Aerts, H. J., Kann, B. H., Mak, R. H., Nelson, T. L., Nguyen, P., Schoenfeld, J. D., … Bitterman, D. S. (2026). An agentic AI system enhances clinical detection of immune-related adverse events in a multi-phase validation study. Nature Cancer. https://doi.org/10.1038/s43018-026-01264-6
Image Credits: AI Generated
DOI: 10.1038/s43018-026-01264-6
Keywords: agentic AI, immune-related adverse events, immune checkpoint inhibitors, large language models, clinical notes, pharmacovigilance, CTCAE grading, electronic health records, human-AI collaboration, randomized crossover study, interannotator agreement, Nature Cancer
News Source: Nathaniel Bowman. (October 29, 2026). Agentic AI speeds and standardizes detection of immune-related adverse events. Scienmag.
