In an analysis examining artificial intelligence applications in medical imaging evaluation, researchers Bo Yang, Lingbo Meng, and Jiageng Shen highlight that assessing large language models requires distinguishing run-to-run repeatability from human expert concordance. Their evaluation examines an investigation by Garello and colleagues that analyzed the methodological quality of cardiac computed tomography and magnetic resonance imaging radiomics studies using both the METhodological RadiomICs Score and the radiomics quality score alongside ChatGPT 5.1 Thinking. While comparing these complementary appraisal instruments identified clinically meaningful discordance between human clinicians and the automated model, Yang and colleagues stress that an additional reliability dimension must be established before such results can be interpreted as supporting scalable, automated radiomics assessment.
The researchers note that Garello and colleagues acknowledged critical study limitations, including the restriction of the language model evaluation to a single model with a fixed prompt structure and an absence of intra-reader analyses. According to Yang and colleagues, these stated limitations specifically address human re-scoring consistency and generalizability across different readers, model architectures, and prompt formulations. However, they emphasize that intra-model repeatability represents an entirely separate methodological question concerning run-to-run variability when the exact same scientific article is evaluated repeatedly under identical prompts and model parameters. Because the original study did not report whether replicate scoring runs were executed or archived, the magnitude and underlying drivers of any run-to-run scoring variation remain unknown, and employing independent temporary sessions with memory disabled cannot substitute for establishing true repeatability.
To demonstrate why these distinct reliability dimensions require separate empirical verification, the authors cite earlier comparative studies evaluating model consistency. In prior work, investigators Mese and Kocak conducted two independent ChatGPT-4o evaluations for each radiomics publication to directly measure intra-model agreement using the radiomics quality score. In an accompanying study focused on the METhodological RadiomICs Score, the same research team compared ChatGPT-4 and NotebookLM against a consensus assessment reached by two radiologists. Yang and colleagues explain that the first experimental approach specifically tested within-model repeatability under stable conditions, whereas the second evaluated cross-model discrepancies and alignment with human specialists, demonstrating that neither reliability dimension can be inferred from a single estimate of human-to-model agreement.
The methodological analysis also raises important questions regarding the statistical metrics used to quantify concordance between automated models and expert readers. In the original evaluation, reported intraclass correlation coefficients were calculated using a two-way random-effects, absolute-agreement, single-measurement specification based on the reporting framework published by Koo and Li. Yang and colleagues point out that this statistical formulation requires a precise definition of the intended rater population, noting it is not self-evident that a fixed, named model and predefined expert panels represent true random samples. Furthermore, because the primary and secondary reader groups may have supplied a consolidated group-level reference rather than separate individual evaluations, clarifying whether expert scores were reached through consensus, mathematical averaging, or item-specific division of labor remains crucial for accurate interpretation.
Because the reported intraclass correlation coefficients compared single model outputs to group-level expert references rather than individual readers, the authors observe that the scope of inference regarding model-assisted appraisal becomes narrowed. If individual item-level scores from participating readers are available, evaluating model concordance against each human evaluator separately would clarify whether observed disagreements stem from subjective rubric ambiguity or inter-rater variation. This analytical separation is especially important given the specific item-level discrepancies documented in the primary research. Garello and colleagues revealed that ChatGPT tended to overestimate METhodological RadiomICs Score items that involve subjective judgment while systematically underestimating radiomics quality score items that demand clinical interpretation, revealing distinct patterns of divergence across different scoring criteria.
Yang and colleagues argue that although generating a single model assessment provides a pragmatic and convenient workflow, any stochastic variation inherent to a single evaluation pass must be rigorously quantified before designating the outputs as reliable. Replicate scoring assessments would allow investigators to separate stable, rubric-related differences from random output fluctuations. They note that identifying stable discrepancies could inform prompt refinement, rubric clarification, evidence-grounded scoring constraints, or targeted expert adjudication. Conversely, if language model outputs exhibit substantial random variation across identical runs, repeated inference and statistical aggregation might mitigate random fluctuations, but such averaging cannot remedy underlying systematic biases at the individual item level.
To strengthen future validation efforts in automated radiomics assessment, the authors recommend that future studies preserve all repeated model evaluation runs and rigorously document specific model versions, prompts, and sampling settings. Whenever technical platforms support it, researchers should record dated model snapshots and fixed random seeds to prevent model updates or configuration changes from being conflated with stochastic variability. Furthermore, validation studies should report total-score intraclass correlation coefficients with confidence intervals accounting for article clustering, match item-level metrics to specific measurement scales, and report agreement separately for single runs and aggregated scores. Yang and colleagues conclude that publishing full run outputs, individual item scores, and supporting source excerpts will provide the necessary evidence to determine whether scalable artificial intelligence appraisal is truly feasible.
Subject of Research: Reliability and repeatability evaluation of large language models in radiomics quality assessment
Article Title: Letter to the Editor: Separating LLM repeatability from expert concordance in radiomics quality assessment
Article References: Yang, B., Meng, L., & Shen, J. (2026). Letter to the Editor: Separating LLM repeatability from expert concordance in radiomics quality assessment. European Radiology Experimental, 10(1), Article 179. https://doi.org/10.1186/s41747-026-00858-0
Image Credits: AI Generated
DOI: 10.1186/s41747-026-00858-0
Keywords: radiomics quality assessment, METhodological RadiomICs Score, radiomics quality score, intra-model repeatability, expert concordance, ChatGPT 5.1 Thinking, cardiac CT, cardiac MRI, intraclass correlation coefficients, run-to-run variability, item-level discrepancies, stochastic output variation, NotebookLM, ChatGPT-4o, two-way random-effects model, error analysis, rater population specification
News Source: Ophelia Keating. (November 13, 2026). Radiomics Quality Appraisal Requires Separating AI Repeatability From Expert Agreement. Scienmag.



