Researchers investigating large language model applications in radiomics quality assessment have clarified that their evaluation measured agreement between artificial intelligence and human experts rather than within-model repeatability. Writing in European Radiology Experimental, the author team addressed methodological distinctions highlighted by Yang, Meng, and Shen regarding how artificial intelligence tools are validated in medical imaging research. The correspondence emphasized that reported intraclass correlation coefficients reflect concordance between model outputs and expert scoring under a specific evaluation configuration, rather than demonstrating consistent performance across repeated evaluations.
In the primary investigation, the investigators evaluated the performance of ChatGPT 5.1 Thinking in applying two standardized radiomics evaluation tools, known as METRICS and the Radiomics Quality Score (RQS). The model evaluated each research article once using standardized prompts in a single-pass framework. To prevent carry-over effects between evaluations, each prompt was deployed in a separate temporary session with the platform memory feature disabled. Because articles were not assessed multiple times, the study was not designed to retrospectively quantify run-to-run variability or comprehensively evaluate internal model reliability.
The authors also clarified the structure of the human expert evaluations used as the comparator in the study. The primary benchmark was established by two human readers with complementary domain expertise who independently scored items within their specific fields. These item-level scores were subsequently combined into a single composite assessment, a method that was likewise applied across secondary reader groups A and B without holding consensus meetings. Consequently, the statistical comparisons evaluated ChatGPT against composite expert evaluations rather than against individual human readers scoring the full set of METRICS and RQS criteria.
Addressing the statistical specifications of the study, the investigators explained their selection of an absolute-agreement, single-measure formulation for calculating intraclass correlation coefficients. This analytical structure treated systematic differences between scores as disagreement and relied on a single score from each assessment setup rather than an averaged score across multiple sessions. The authors noted that because human reader groups, model choice, and prompting strategies were fixed, the resulting coefficients reflect agreement between those specific configurations rather than estimates that can be broadly generalized to other readers, alternative language models, or different prompting strategies.
Technical constraints associated with the testing environment also shaped the scope of the findings. The investigators accessed ChatGPT 5.1 Thinking through its web application interface on December 8, 2025, where sampling parameters such as temperature were not user-configurable. To advance reproducibility and distinguish random stochastic variability from systematic scoring discrepancies, the authors recommended that future validation studies preserve run-level outputs across repeated assessments and document model parameters whenever platform access permits.
The author team concluded that item-level discrepancies identified in their study represent descriptive patterns of single-pass assessments rather than stable, persistent biases across repeated model executions. While highlighting that repeated assessments will be necessary to evaluate intra-model repeatability in future work, the researchers maintained their initial conclusion: ChatGPT 5.1 Thinking shows potential to assist in radiomics quality scoring, but ongoing discrepancies with composite expert evaluations continue to warrant caution.
Regarding the mathematical modeling choices, the researchers explained that their intraclass correlation coefficient calculations yield identical numerical results under alternative statistical formulations. Specifically, when applied to their collected data, a two-way mixed-effects, absolute-agreement, single-measure model produces the exact same numerical point estimate as a two-way random-effects formulation. The author team emphasized that selecting between these two formulations affects the theoretical interpretation of generalizability rather than altering the reported numerical concordance values.
Subject of Research: Evaluation of large language model repeatability and agreement in radiomics quality assessment
Article Title: Reply to the Letter to the Editor: Separating LLM repeatability from expert concordance in radiomics quality assessment
Article References: Garello, L. F., Giannini, V., Gatti, M., Defeudis, A., Cafaro, D., Nicoletti, G., Culasso, N. C., Veltri, A., Cuocolo, R., & Balbi, M. (2026). Reply to the Letter to the Editor: Separating LLM repeatability from expert concordance in radiomics quality assessment. European Radiology Experimental, 10(1), Article 180. https://doi.org/10.1186/s41747-026-00859-z
Image Credits: AI Generated
DOI: 10.1186/s41747-026-00859-z
Keywords: radiomics quality assessment, ChatGPT 5.1 Thinking, intraclass correlation coefficient, METRICS score, Radiomics Quality Score, intra-model repeatability, human-LLM agreement, single-pass evaluation, composite expert assessment, large language models, two-way mixed-effects model, absolute agreement, model inference conditions, run-to-run variability, European Radiology Experimental, stochastic variability, medical imaging quality, standardized prompting
News Source: Ophelia Keating. (November 13, 2026). ChatGPT Assessment in Radiomics Reflects Human Concordance Rather Than Internal Repeatability. Scienmag.



