• HOME
  • NEWS
  • EXPLORE
    • CAREER
      • Companies
      • Jobs
    • EVENTS
    • iGEM
      • News
      • Team
    • PHOTOS
    • VIDEO
    • WIKI
  • BLOG
  • COMMUNITY
    • FACEBOOK
    • INSTAGRAM
    • TWITTER
Thursday, November 19, 2026
BIOENGINEER.ORG
No Result
View All Result
  • Login
  • HOME
  • NEWS
  • EXPLORE
    • CAREER
      • Companies
      • Jobs
        • Lecturer
        • PhD Studentship
        • Postdoc
        • Research Assistant
    • EVENTS
    • iGEM
      • News
      • Team
    • PHOTOS
    • VIDEO
    • WIKI
  • BLOG
  • COMMUNITY
    • FACEBOOK
    • INSTAGRAM
    • TWITTER
  • HOME
  • NEWS
  • EXPLORE
    • CAREER
      • Companies
      • Jobs
        • Lecturer
        • PhD Studentship
        • Postdoc
        • Research Assistant
    • EVENTS
    • iGEM
      • News
      • Team
    • PHOTOS
    • VIDEO
    • WIKI
  • BLOG
  • COMMUNITY
    • FACEBOOK
    • INSTAGRAM
    • TWITTER
No Result
View All Result
Bioengineer.org
No Result
View All Result
Home NEWS Science News Health

New Safety Framework Adapts Medical Failure Analysis to Clinical Language Models

by
November 12, 2026
in Health, Technology
Reading Time: 4 mins read
0
New Safety Framework Adapts Medical Failure Analysis to Clinical Language Models

New Safety Framework Adapts Medical Failure Analysis to Clinical Language Models

Share on FacebookShare on TwitterShare on LinkedinShare on RedditShare on Telegram

Researchers have developed and refined an evaluation framework adapting Failure Mode, Effects, and Criticality Analysis to identify and characterize potential patient safety risks in clinical summaries generated by large language models. The methodological pilot study established an operational taxonomy comprising 14 failure modes across six categories, providing a structured approach to generate safety evidence before artificial intelligence tools are deployed in clinical environments.

Claims that large language models improve healthcare workflows have outpaced the empirical evidence required to verify their safety, leaving a shortage of structured methods to evaluate outputs before they can be trusted in practice. While Failure Mode, Effects, and Criticality Analysis represents an established methodology in medicine for generating proportional pre-deployment safety assessments, the technique had not previously been adapted to generative artificial intelligence applications.

To formulate the framework, an interdisciplinary panel of eight experts participated in a two-round iterative consensus process. Starting from an initial pool of 78 candidate failure modes, the panel consolidated the taxonomy into 14 distinct failure modes across six categories and adapted standard assessment dimensions—specifically occurrence, severity, and detectability—into operational five-point ordinal scales.

The research team subsequently refined the framework by evaluating 36 clinical summaries produced from Geneva University Hospitals data using an open-weight large language model, GPT-OSS 120B. Across two annotation rounds, three clinical reviewers analyzed the generated texts to evaluate the operational criteria and refine scoring definitions.

Exploratory secondary analyses indicated that inter-rater agreement among the reviewers improved between the two rounds. The evaluations achieved moderate-to-substantial agreement for identifying failure modes and reached good agreement for severity and detectability scoring, while an adapted System Usability Scale evaluation yielded a score of 79.2 out of 100, reflecting good internal usability.

The study authors noted that occurrence rates and overall risk priority were deliberately not estimated during the pilot phase, as the sample was designed specifically for methodological refinement. Consequently, the researchers concluded that the resulting taxonomy will require further testing through large-scale empirical validation studies.

The investigation was structured around addressing a critical gap in clinical artificial intelligence safety protocols, given that conventional healthcare safety methods had not previously been translated to generative models. Standard Failure Mode, Effects, and Criticality Analysis methodology focuses on evaluating potential points of system breakdown through predefined dimensions, which the interdisciplinary team converted into five-point ordinal measurement tools to reflect the nuances of clinical text generated by machine learning systems. In constructing the scoring framework, the eight-member panel evaluated the suitability of each metric specifically for the operational review of automated documentation.

During the practical refinement phase, clinical texts were processed using data sourced from Geneva University Hospitals, providing a realistic basis for evaluating how generative systems behave under institutional conditions. The 36 clinical summaries produced by the open-weight GPT-OSS 120B model allowed the three clinical reviewers to test the practical boundaries of the taxonomy across two dedicated rounds of annotation. This structured review sequence enabled the team to evaluate whether the operational definitions for severity and detectability scales could be consistently applied across different evaluators examining artificial intelligence summaries.

In the exploratory secondary evaluations conducted to measure reviewer consistency, inter-rater reliability was tracked specifically across three analytical levels: failure-mode determination, severity grading, and detectability scoring. The improvement observed between the first and second rounds indicated that reviewer agreement progressed from initial baseline levels to moderate-to-substantial concordance for identifying failure modes. Simultaneously, consistency in evaluating severity and detectability reached good agreement, providing preliminary evidence regarding the framework’s internal reliability when utilized by trained clinical reviewers.

The evaluation of internal usability through an adapted System Usability Scale further contextualized how clinical evaluators interact with the multi-step assessment tool. The resulting score of 79.2 out of 100 placed the framework within the scale’s category of good internal usability, suggesting that the 14-mode taxonomy across six operational categories remains manageable during structured safety reviews despite the complexity of evaluating model-generated documentation.

Because the pilot study prioritized refining the assessment structure rather than establishing baseline performance metrics, the investigators purposefully omitted evaluations of overall risk priority numbers or absolute occurrence rates. The authors emphasized that because the pilot sample size of 36 summaries was specifically curated for methodological development, comprehensive empirical validation across larger and more diverse clinical datasets will be necessary before the taxonomy can support standardized pre-deployment risk benchmarking in healthcare settings.

Subject of Research: A Failure Mode, Effects, and Criticality Analysis framework for clinical language model summaries

Article Title: Development and refinement of a FMECA framework for identifying patient safety risks in LLM-generated clinical summaries: a methodological pilot study

Article References: Bednarczyk, L., Zaghir, J., Ehrsam, J., Tcherepanova, M., Skalafouris, C., Gariani, K., Geslin, C., Rivara, C.-B., Bonnabry, P., Goldman, J.-P., Gosetto, L., Dubos, R., Bellazzi, R., Bjelogrlic, M., Gaudet-Blavignac, C., & Lovis, C. (2026). Development and refinement of a FMECA framework for identifying patient safety risks in LLM-generated clinical summaries: a methodological pilot study. BMC Medical Informatics and Decision Making. https://doi.org/10.1186/s12911-026-03969-1

Image Credits: AI Generated

DOI: 10.1186/s12911-026-03969-1

Keywords: Failure Mode Effects and Criticality Analysis, FMECA, large language models, clinical summaries, patient safety, medical artificial intelligence, generative AI, GPT-OSS 120B, Geneva University Hospitals, inter-rater reliability, System Usability Scale, pre-deployment safety, failure mode taxonomy, expert consensus, clinical text summarization, risk assessment

News Source: Denise Maddox. (November 12, 2026). New Safety Framework Adapts Medical Failure Analysis to Clinical Language Models. Scienmag.

Tags: Clinical Summariesclinical text summarizationExpert consensusFailure Mode Effects and Criticality Analysisfailure mode taxonomyFMECAgenerative AIGeneva University HospitalsGPT-OSS 120Binter-rater reliabilityLarge Language Modelsmedical artificial intelligencePatient Safetypre-deployment safetyrisk assessmentSystem Usability Scale
Share12Tweet7Share2ShareShareShare1

Related Posts

Scoping Review Finds Only Four Studies on Maxillofacial Trauma in LGBTQIA+ Violence

Scoping Review Finds Only Four Studies on Maxillofacial Trauma in LGBTQIA+ Violence

November 19, 2026
German Society of Neurology Updates Guidelines for HIV-Associated Neurological Disorders

German Society of Neurology Updates Guidelines for HIV-Associated Neurological Disorders

November 19, 2026

Surgical Risk Prediction Models Show Disparities and Sparse Auditing Across Patient Groups

November 19, 2026

Winter Migratory Ducks Carry Diverse Antibiotic Resistance Genes Across Japan

November 19, 2026

POPULAR NEWS

  • Municipal Leaders Rely on Tech Champions Lacking Training and Compensation

    Municipal Leaders Rely on Tech Champions Lacking Training and Compensation

    29 shares
    Share 12 Tweet 7
  • Virtual Reality Cognitive Games Show Preliminary Promise for Stroke Recovery Outcomes

    29 shares
    Share 12 Tweet 7
  • Alzheimer’s Risk Gene SORL1 Deficiency Impairs Mitochondrial Function in Human Microglia

    29 shares
    Share 12 Tweet 7
  • Optimized Streptozotocin Delivery Regimen Improves Sporadic Alzheimer’s Disease Modeling in Rodents

    29 shares
    Share 12 Tweet 7

About

We bring you the latest biotechnology news from best research centers and universities around the world. Check our website.

Follow us

Recent News

Scoping Review Finds Only Four Studies on Maxillofacial Trauma in LGBTQIA+ Violence

German Society of Neurology Updates Guidelines for HIV-Associated Neurological Disorders

Bath Researchers Secure £1.7 Million Investment to Advance Acrylic Plastic Recycling

Subscribe to Blog via Email

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Join 88 other subscribers
  • Contact Us

Bioengineer.org © Copyright 2023 All Rights Reserved.

Welcome Back!

Login to your account below

Forgotten Password?

Retrieve your password

Please enter your username or email address to reset your password.

Log In
No Result
View All Result
  • Homepages
    • Home Page 1
    • Home Page 2
  • News
  • National
  • Business
  • Health
  • Lifestyle
  • Science

Bioengineer.org © Copyright 2023 All Rights Reserved.