• HOME
  • NEWS
  • EXPLORE
    • CAREER
      • Companies
      • Jobs
    • EVENTS
    • iGEM
      • News
      • Team
    • PHOTOS
    • VIDEO
    • WIKI
  • BLOG
  • COMMUNITY
    • FACEBOOK
    • INSTAGRAM
    • TWITTER
Monday, November 16, 2026
BIOENGINEER.ORG
No Result
View All Result
  • Login
  • HOME
  • NEWS
  • EXPLORE
    • CAREER
      • Companies
      • Jobs
        • Lecturer
        • PhD Studentship
        • Postdoc
        • Research Assistant
    • EVENTS
    • iGEM
      • News
      • Team
    • PHOTOS
    • VIDEO
    • WIKI
  • BLOG
  • COMMUNITY
    • FACEBOOK
    • INSTAGRAM
    • TWITTER
  • HOME
  • NEWS
  • EXPLORE
    • CAREER
      • Companies
      • Jobs
        • Lecturer
        • PhD Studentship
        • Postdoc
        • Research Assistant
    • EVENTS
    • iGEM
      • News
      • Team
    • PHOTOS
    • VIDEO
    • WIKI
  • BLOG
  • COMMUNITY
    • FACEBOOK
    • INSTAGRAM
    • TWITTER
No Result
View All Result
Bioengineer.org
No Result
View All Result
Home NEWS Science News Health

Radiomics Quality Appraisal Requires Separating AI Repeatability From Expert Agreement

by
November 13, 2026
in Health
Reading Time: 4 mins read
0
Radiomics Quality Appraisal Requires Separating AI Repeatability From Expert Agreement

Radiomics Quality Appraisal Requires Separating AI Repeatability From Expert Agreement

Share on FacebookShare on TwitterShare on LinkedinShare on RedditShare on Telegram

In an analysis examining artificial intelligence applications in medical imaging evaluation, researchers Bo Yang, Lingbo Meng, and Jiageng Shen highlight that assessing large language models requires distinguishing run-to-run repeatability from human expert concordance. Their evaluation examines an investigation by Garello and colleagues that analyzed the methodological quality of cardiac computed tomography and magnetic resonance imaging radiomics studies using both the METhodological RadiomICs Score and the radiomics quality score alongside ChatGPT 5.1 Thinking. While comparing these complementary appraisal instruments identified clinically meaningful discordance between human clinicians and the automated model, Yang and colleagues stress that an additional reliability dimension must be established before such results can be interpreted as supporting scalable, automated radiomics assessment.

The researchers note that Garello and colleagues acknowledged critical study limitations, including the restriction of the language model evaluation to a single model with a fixed prompt structure and an absence of intra-reader analyses. According to Yang and colleagues, these stated limitations specifically address human re-scoring consistency and generalizability across different readers, model architectures, and prompt formulations. However, they emphasize that intra-model repeatability represents an entirely separate methodological question concerning run-to-run variability when the exact same scientific article is evaluated repeatedly under identical prompts and model parameters. Because the original study did not report whether replicate scoring runs were executed or archived, the magnitude and underlying drivers of any run-to-run scoring variation remain unknown, and employing independent temporary sessions with memory disabled cannot substitute for establishing true repeatability.

To demonstrate why these distinct reliability dimensions require separate empirical verification, the authors cite earlier comparative studies evaluating model consistency. In prior work, investigators Mese and Kocak conducted two independent ChatGPT-4o evaluations for each radiomics publication to directly measure intra-model agreement using the radiomics quality score. In an accompanying study focused on the METhodological RadiomICs Score, the same research team compared ChatGPT-4 and NotebookLM against a consensus assessment reached by two radiologists. Yang and colleagues explain that the first experimental approach specifically tested within-model repeatability under stable conditions, whereas the second evaluated cross-model discrepancies and alignment with human specialists, demonstrating that neither reliability dimension can be inferred from a single estimate of human-to-model agreement.

The methodological analysis also raises important questions regarding the statistical metrics used to quantify concordance between automated models and expert readers. In the original evaluation, reported intraclass correlation coefficients were calculated using a two-way random-effects, absolute-agreement, single-measurement specification based on the reporting framework published by Koo and Li. Yang and colleagues point out that this statistical formulation requires a precise definition of the intended rater population, noting it is not self-evident that a fixed, named model and predefined expert panels represent true random samples. Furthermore, because the primary and secondary reader groups may have supplied a consolidated group-level reference rather than separate individual evaluations, clarifying whether expert scores were reached through consensus, mathematical averaging, or item-specific division of labor remains crucial for accurate interpretation.

Because the reported intraclass correlation coefficients compared single model outputs to group-level expert references rather than individual readers, the authors observe that the scope of inference regarding model-assisted appraisal becomes narrowed. If individual item-level scores from participating readers are available, evaluating model concordance against each human evaluator separately would clarify whether observed disagreements stem from subjective rubric ambiguity or inter-rater variation. This analytical separation is especially important given the specific item-level discrepancies documented in the primary research. Garello and colleagues revealed that ChatGPT tended to overestimate METhodological RadiomICs Score items that involve subjective judgment while systematically underestimating radiomics quality score items that demand clinical interpretation, revealing distinct patterns of divergence across different scoring criteria.

Yang and colleagues argue that although generating a single model assessment provides a pragmatic and convenient workflow, any stochastic variation inherent to a single evaluation pass must be rigorously quantified before designating the outputs as reliable. Replicate scoring assessments would allow investigators to separate stable, rubric-related differences from random output fluctuations. They note that identifying stable discrepancies could inform prompt refinement, rubric clarification, evidence-grounded scoring constraints, or targeted expert adjudication. Conversely, if language model outputs exhibit substantial random variation across identical runs, repeated inference and statistical aggregation might mitigate random fluctuations, but such averaging cannot remedy underlying systematic biases at the individual item level.

To strengthen future validation efforts in automated radiomics assessment, the authors recommend that future studies preserve all repeated model evaluation runs and rigorously document specific model versions, prompts, and sampling settings. Whenever technical platforms support it, researchers should record dated model snapshots and fixed random seeds to prevent model updates or configuration changes from being conflated with stochastic variability. Furthermore, validation studies should report total-score intraclass correlation coefficients with confidence intervals accounting for article clustering, match item-level metrics to specific measurement scales, and report agreement separately for single runs and aggregated scores. Yang and colleagues conclude that publishing full run outputs, individual item scores, and supporting source excerpts will provide the necessary evidence to determine whether scalable artificial intelligence appraisal is truly feasible.

Subject of Research: Reliability and repeatability evaluation of large language models in radiomics quality assessment

Article Title: Letter to the Editor: Separating LLM repeatability from expert concordance in radiomics quality assessment

Article References: Yang, B., Meng, L., & Shen, J. (2026). Letter to the Editor: Separating LLM repeatability from expert concordance in radiomics quality assessment. European Radiology Experimental, 10(1), Article 179. https://doi.org/10.1186/s41747-026-00858-0

Image Credits: AI Generated

DOI: 10.1186/s41747-026-00858-0

Keywords: radiomics quality assessment, METhodological RadiomICs Score, radiomics quality score, intra-model repeatability, expert concordance, ChatGPT 5.1 Thinking, cardiac CT, cardiac MRI, intraclass correlation coefficients, run-to-run variability, item-level discrepancies, stochastic output variation, NotebookLM, ChatGPT-4o, two-way random-effects model, error analysis, rater population specification

News Source: Ophelia Keating. (November 13, 2026). Radiomics Quality Appraisal Requires Separating AI Repeatability From Expert Agreement. Scienmag.

Tags: cardiac CTcardiac MRIChatGPT 5.1 ThinkingChatGPT-4oerror analysisexpert concordanceintra-model repeatabilityintraclass correlation coefficientsitem-level discrepanciesMEThodological RadiomICs ScoreNotebookLMradiomics quality assessmentRadiomics Quality Scorerater population specificationrun-to-run variabilitystochastic output variationtwo-way random-effects model
Share12Tweet7Share2ShareShareShare1

Related Posts

Sports Ethicists Urge Strict Limits on Performance Enhancing Swimming and Running Technologies

Sports Ethicists Urge Strict Limits on Performance Enhancing Swimming and Running Technologies

November 16, 2026
Case Study Links Antipsychotic Drug Clozapine to Rare Colon Eosinophilic Cryptitis

Case Study Links Antipsychotic Drug Clozapine to Rare Colon Eosinophilic Cryptitis

November 16, 2026

Over Seventy Percent of Neuropsychiatric Outpatients Show Low Treatment Adherence

November 16, 2026

Mapping Dental Clinic Clusters Using Mobile Geospatial Technology in India

November 16, 2026

POPULAR NEWS

  • Strengthening Patient Safety Culture in Fragile African Health Systems

    Strengthening Patient Safety Culture in Fragile African Health Systems

    29 shares
    Share 12 Tweet 7
  • Platform AI Assistance Linked to Higher Online Medical Consultation Service Quality

    29 shares
    Share 12 Tweet 7
  • Somali Pastoral Herd Sales Depend on Births and Theft Alongside Herd Size

    29 shares
    Share 12 Tweet 7
  • Silicon Oxide Composite Anode Retains High Capacity Over Three Hundred Cycles

    29 shares
    Share 12 Tweet 7

About

We bring you the latest biotechnology news from best research centers and universities around the world. Check our website.

Follow us

Recent News

Sports Ethicists Urge Strict Limits on Performance Enhancing Swimming and Running Technologies

Case Study Links Antipsychotic Drug Clozapine to Rare Colon Eosinophilic Cryptitis

OneNet Connector Enables Trusted Interoperable Energy Data Exchange

Subscribe to Blog via Email

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Join 89 other subscribers
  • Contact Us

Bioengineer.org © Copyright 2023 All Rights Reserved.

Welcome Back!

Login to your account below

Forgotten Password?

Retrieve your password

Please enter your username or email address to reset your password.

Log In
No Result
View All Result
  • Homepages
    • Home Page 1
    • Home Page 2
  • News
  • National
  • Business
  • Health
  • Lifestyle
  • Science

Bioengineer.org © Copyright 2023 All Rights Reserved.