Three flagship commercial multimodal large language models failed to reach agreement levels sufficient for autonomous extraction of clinically relevant breast pathology features, according to a single-center retrospective paired concordance study comparing Claude Sonnet 4.6, Gemini 2.5 Pro and GPT-5.5 against a board-certified breast pathologist reference in 427 invasive breast cancer cases. The study, published in Breast Cancer Research, examined whether paid-access AI systems that read hematoxylin and eosin images could reproduce specialist-level reporting of features such as Nottingham grade and receptor-defined subtype, which guide breast cancer treatment but depend on specialist pathologist access that is limited where the workforce is scarce.
Each model received the same overview and 16 H&E tissue tiles per case and the same prompt. The researchers assessed item-level concordance for clinically relevant features, compared raw and chance-corrected agreement, and characterized the direction of each model’s errors. In 111 cases, two pathologists blinded to the report re-read the submitted slide, and the image packets given to the models were reviewed separately, establishing slide-level and packet-level references and a within-study interobserver benchmark.
No model achieved agreement sufficient to support autonomous extraction of any feature. On blinded review of the image packets in 110 cases, lymphovascular invasion was visible in none and perineural invasion in one; Claude and GPT-5.5 reported neither, whereas Gemini reported lymphovascular invasion in 26 packets, none supported by the images. Invasive carcinoma was absent from five packets, for which a Nottingham grade was nevertheless returned in 14 of 15 model-case pairs.
Claude ranked highest by raw histological-type concordance at 87.4 percent, yet classified every lobular and micropapillary carcinoma as no-special-type, a pattern that reversed the ranking in the class-wise sensitivity analysis. In an exploratory negative-control analysis against IHC-defined receptor classes, HER2-positive sensitivity was at most 3.6 percent across all three models, the expected floor for H&E-only input, since receptor status is normally determined by immunohistochemistry rather than by routine stained slides alone.
Two-reader agreement on the submitted slides was moderate, with a quadratic-weighted kappa for grade of 0.51. Model agreement with the report for grade ranged from 0.27 to 0.42, a range that overlapped with the second pathologist’s agreement of 0.36. The authors note that this overlap does not establish model equivalence to specialist review, because the models and the second reader were assessed against different reference structures within the study.
Under the same prompt, each model showed a systematic error direction, including positive calls for findings absent from the input. Limited secondary analyses in individual models, covering prompt variants, reasoning-effort settings, and an alternative higher-tier model from the same vendor, did not eliminate these patterns. Whether they reflect model priors or prompt-model interaction was not resolved by the study.
The findings do not support autonomous use of these models for breast pathology reporting. Any assistive role would require prospective evaluation with tumor-aware image sampling and pathologist review, according to the study’s conclusions. The work was released as an open-access research article on 4 November 2026, shared early to provide faster access to peer-reviewed, accepted research, with a version subject to further edits before the final Version of Record.
The study design addressed a gap in prior pathology evaluations of commercial MLLMs, which had addressed only coarse tasks. By using a paired concordance framework with blinded re-reading and packet-level review, the researchers could separate what was actually visible in the image input from what the models reported, revealing that some models produced positive findings with no support in the images they were given.
Because the evaluation was retrospective and single-center, the results characterize these three specific models under the tested prompt and image-sampling conditions rather than establishing performance limits for all AI systems or all clinical settings. The authors frame the systematic error directions as a reason for caution: the same prompt produced model-specific failure patterns, meaning that improving one model’s weakness does not predict another model’s behavior, and that safe deployment, if any, would depend on controlled image selection and human oversight.
Subject of Research: Systematic model-specific errors of commercial multimodal large language models in breast cancer pathology reporting
Article Title: Commercial multimodal large language models show systematic model-specific errors in breast cancer pathology reporting: a paired concordance study
Article References: Jun, S.-Y., Kim, S., An, S., & Kang, Y.-J. (2026). Commercial multimodal large language models show systematic model-specific errors in breast cancer pathology reporting: a paired concordance study. Breast Cancer Research. https://doi.org/10.1186/s13058-026-02413-6
Image Credits: AI Generated
DOI: 10.1186/s13058-026-02413-6
Keywords: breast cancer, pathology, multimodal large language models, paired concordance study, Nottingham grade, lymphovascular invasion, HER2, histological type, H&E images, Claude, Gemini, GPT-5.5
News Source: Nathaniel Bowman. (November 6, 2026). AI models show systematic errors in breast cancer pathology reporting. Scienmag.
