• HOME
  • NEWS
  • EXPLORE
    • CAREER
      • Companies
      • Jobs
    • EVENTS
    • iGEM
      • News
      • Team
    • PHOTOS
    • VIDEO
    • WIKI
  • BLOG
  • COMMUNITY
    • FACEBOOK
    • INSTAGRAM
    • TWITTER
Tuesday, November 17, 2026
BIOENGINEER.ORG
No Result
View All Result
  • Login
  • HOME
  • NEWS
  • EXPLORE
    • CAREER
      • Companies
      • Jobs
        • Lecturer
        • PhD Studentship
        • Postdoc
        • Research Assistant
    • EVENTS
    • iGEM
      • News
      • Team
    • PHOTOS
    • VIDEO
    • WIKI
  • BLOG
  • COMMUNITY
    • FACEBOOK
    • INSTAGRAM
    • TWITTER
  • HOME
  • NEWS
  • EXPLORE
    • CAREER
      • Companies
      • Jobs
        • Lecturer
        • PhD Studentship
        • Postdoc
        • Research Assistant
    • EVENTS
    • iGEM
      • News
      • Team
    • PHOTOS
    • VIDEO
    • WIKI
  • BLOG
  • COMMUNITY
    • FACEBOOK
    • INSTAGRAM
    • TWITTER
No Result
View All Result
Bioengineer.org
No Result
View All Result
Home NEWS Science News Health

Study Finds AI Chatbots Fail Reading Benchmarks for Melanoma Education

by
November 17, 2026
in Health
Reading Time: 5 mins read
0
Study Finds AI Chatbots Fail Reading Benchmarks for Melanoma Education

Study Finds AI Chatbots Fail Reading Benchmarks for Melanoma Education

Share on FacebookShare on TwitterShare on LinkedinShare on RedditShare on Telegram

A cross-sectional comparative investigation published in Frontiers in Public Health has revealed that while contemporary artificial intelligence chatbots can answer melanoma-related medical examination questions with remarkable accuracy, their written explanations remain too linguistically complex for the general public and consistently lack foundational transparency regarding their sources. Conducted by an interdisciplinary team of clinical researchers from Central South University and the University of South China, the study evaluated four leading conversational platforms: ChatGPT-5.6, Gemini 3.5 Flash, Claude Sonnet 5, and DeepSeek-V4. The researchers discovered that despite differences in medical knowledge precision, not a single evaluated model generated patient-oriented guidance that conformed to the sixth-grade reading level standard universally recommended for consumer health information.

The clinical motivation behind the evaluation stems from the rapid surge in consumer reliance on generative artificial intelligence for personal medical inquiries alongside the growing global burden of melanoma. According to GLOBOCAN 2022 epidemiological data cited by the investigators, approximately 330,000 newly diagnosed melanoma cases and nearly 60,000 associated deaths occurred worldwide in 2022, with international forecasts projecting an increase to roughly 510,000 annual cases and 96,000 fatalities by 2040 if current incidence trends continue unchecked. However, previous assessments of digital resources have demonstrated that patient-facing melanoma websites consistently display wide variations in clinical reliability while averaging a college-level reading grade of 12.6, far exceeding recommended thresholds.

To rigorously determine whether conversational generative models solve or replicate these health communication barriers, the investigators established an objective two-part methodology that combined standardized professional knowledge testing with evaluations of authentic public inquiries. In the first phase, response validity was tested using a curated bank of 200 single-best-answer multiple-choice questions. This diagnostic test suite comprised 30 questions derived from the Medical Question Answering dataset, 137 questions from the Medical Multiple-Choice Question Answering dataset, and 33 questions drawn from Chinese national medical qualification examinations. The collection encompassed etiology, pathological hallmarks, anatomical presentations, and therapeutic strategies, with all answer keys independently verified by two clinicians against standard medical references without discrepancy.

To evaluate the chatbots under strict, reproducible conditions, each of the 200 examination questions was submitted independently across three separate trials to each chatbot through its standard official web interface, utilizing default parameter configurations without active web browsing or retrieval augmentation. Across these 600 total query administrations per platform, ChatGPT-5.6 achieved the highest mean accuracy at 94.0 percent with a standard deviation of 0.5 percent, corresponding to an error rate of 6.0 percent. Gemini 3.5 Flash followed closely with an overall mean accuracy of 92.5 percent and an error rate of 7.5 percent. Statistical analysis via one-way analysis of variance and Tukey multiple-comparison testing revealed that the 1.50-percentage-point difference between ChatGPT-5.6 and Gemini 3.5 Flash was not statistically significant. In contrast, Claude Sonnet 5 attained an accuracy of 87.2 percent, while DeepSeek-V4 scored lowest at 79.3 percent, demonstrating statistically significant performance gaps compared to the leading models.

In the second phase of the investigation, the research team designed an evaluation battery reflecting genuine consumer behavior rather than theoretical academic inquiries. The authors analyzed global search patterns via Google Trends over a five-year window spanning July 8, 2021, to July 8, 2026. The search data showed a progressive upward trajectory in worldwide interest that culminated in peak activity in June 2026, with Australia, New Zealand, Italy, the United States, the United Kingdom, and Ireland recording the highest relative search interest. Synthesizing the 25 most frequent international search queries with real-world plastic surgery clinical practice, a specialist panel developed 15 core patient-oriented questions spanning disease awareness, subungual and acral anatomical subtypes, diagnostic procedures such as biopsy, and systemic therapeutics for metastatic disease.

The 15 public education questions were submitted verbatim in English to all four systems, and the resulting narrative responses were assessed by two independent plastic surgeons using four validated reliability and quality instruments: DISCERN, the 20-item Ensuring Quality Information for Patients tool, the Global Quality Scale, and the Journal of the American Medical Association benchmark criteria. Inter-rater reliability between the surgical reviewers prior to consensus was exceptionally high, yielding intraclass correlation coefficients of 0.971 for DISCERN, 0.984 for EQIP, and 1.000 for JAMA benchmarks, alongside a moderate agreement of 0.712 for the Global Quality Scale. The resulting evaluations highlighted marked disparities between factual accuracy and communicative quality across the different AI platforms.

On qualitative scoring, ChatGPT-5.6 achieved the highest mean values across all primary reliability measures, recording a DISCERN score of 52.07 out of 80, an EQIP score of 81.67 percent, and a Global Quality Scale rating of 4.40 out of 5. DeepSeek-V4 scored 44.33 on DISCERN, 80.67 percent on EQIP, and 4.00 on GQS, demonstrating that clear thematic organization can yield high consumer communication marks despite lower benchmark accuracy. Claude Sonnet 5 achieved corresponding scores of 44.87, 72.33 percent, and 4.13, while Gemini 3.5 Flash recorded 39.73 on DISCERN, 75.33 percent on EQIP, and 3.87 on GQS. However, all models performed poorly regarding accountability: responses from ChatGPT-5.6, DeepSeek-V4, and Gemini 3.5 Flash satisfied only one of the four JAMA criteria, while Claude Sonnet 5 satisfied none, as none consistently disclosed authorship, evidentiary dates, or verifiable clinical source attributions.

The assessment of linguistic complexity, conducted across six standard readability indices including the Automated Readability Index, Coleman-Liau Index, Flesch-Kincaid Grade Level, Flesch Reading Ease Score, Gunning Fog Index, and Simple Measure of Gobbledygook, documented a pervasive failure to meet consumer health literacy targets. Across all four models, every mean grade-level index substantially exceeded the benchmark value of 6.0, and every mean reading ease score remained below the recommended target range of 80 to 90.

Gemini 3.5 Flash exhibited the most favorable overall readability profile among the models, generating the lowest mean Automated Readability Index of 10.16, Coleman-Liau Index of 9.99, Gunning Fog Index of 10.43, and SMOG score of 9.23, along with the highest Flesch Reading Ease score of 56.00. DeepSeek-V4 recorded the lowest mean Flesch-Kincaid Grade Level at 9.65. Conversely, ChatGPT-5.6, which demonstrated the most thorough treatment discussions and highest diagnostic accuracy, produced the most difficult text, registering the highest Automated Readability Index at 11.35, Coleman-Liau at 11.59, Gunning Fog at 11.72, and the lowest Flesch Reading Ease score of 47.93.

Beyond mathematical formulas, an exploratory surgical review of chatbot explanations regarding diagnostic workups and surgical staging uncovered clinically notable omissions. While all four models appropriately endorsed excisional biopsy as the primary diagnostic approach, explanations of biopsy depth and Breslow thickness were inconsistent, and margins were not reliably connected to tumor thickness. Furthermore, in response to queries regarding positive sentinel lymph node biopsies, DeepSeek-V4 suggested that completion lymph node dissection remained routine, failing to reflect modern guidelines that prioritize nodal ultrasound surveillance over immediate radical clearance. Additionally, none of the models addressed reconstructive management, omitting skin flaps or grafts needed for large surgical defects.

The authors concluded that while current large language models possess extensive melanoma medical knowledge, their raw outputs cannot be considered direct replacements for personalized clinical consultations. The research team noted several practical limitations in their study, including the cross-sectional evaluation of specific software versions, lack of user demographics in Google Trends data, and the restriction of readability formulas to English text. Nevertheless, the findings demonstrate that until automated models incorporate system-level controls for reading level, robust source citation, and explicit statements regarding clinical uncertainty, conversational artificial intelligence should function solely as a supplementary educational aid under professional medical guidance.

Subject of Research: Comparative evaluation of large language model chatbots for melanoma public health education

Article Title: Validity, reliability, and readability of large language model chatbots for melanoma public education: a cross-sectional comparative study

Article References: Wei, Y., Yu, T., Tang, D., Xiong, T., Li, L., & Liao, J. (2026). Validity, reliability, and readability of large language model chatbots for melanoma public education: a cross-sectional comparative study. Frontiers in Public Health, 14, Article 2000750. https://doi.org/10.3389/fpubh.2026.2000750

Image Credits: AI Generated

DOI: 10.3389/fpubh.2026.2000750

Keywords: artificial intelligence, melanoma education, large language models, health literacy, readability, ChatGPT, Gemini, Claude, DeepSeek, DISCERN, EQIP, public health, dermatology, clinical communication, medical information quality, oncology

News Source: Nathaniel Bowman. (November 17, 2026). Study Finds AI Chatbots Fail Reading Benchmarks for Melanoma Education. Scienmag.

Tags: Artificial IntelligenceChatGPTClaudeclinical communicationDeepSeekDermatologyDISCERNEQIPGeminihealth literacyLarge Language Modelsmedical information qualitymelanoma educationOncologyPublic Healthreadability
Share12Tweet7Share2ShareShareShare1

Related Posts

Custom-made prostheses provide functional shoulder reconstruction after scapular tumor resections

Custom-made prostheses provide functional shoulder reconstruction after scapular tumor resections

November 17, 2026
Subtle BRCA2 Gene Alterations Link to Moderate Breast Cancer Risks

Subtle BRCA2 Gene Alterations Link to Moderate Breast Cancer Risks

November 17, 2026

Review Maps Multi-System Stress Mechanisms and Clinical Limits in Takotsubo Syndrome

November 17, 2026

Talar Osteochondral Lesions Found in Most Surgically Treated Trimalleolar Ankle Patients

November 17, 2026

POPULAR NEWS

  • High Prevalence of Gender-Based Violence Follows Armed Conflict in Southern Tigray

    High Prevalence of Gender-Based Violence Follows Armed Conflict in Southern Tigray

    29 shares
    Share 12 Tweet 7
  • Air Pollutants Tied to Adverse Sleep Outcomes Amid Notable Evidence Uncertainty

    29 shares
    Share 12 Tweet 7
  • Network Analysis Identifies Core Positive Development Attributes in Rural Chinese Adolescents

    29 shares
    Share 12 Tweet 7
  • China Silver Economy Policies Show Good Overall Consistency but Uneven Regional Breadth

    29 shares
    Share 12 Tweet 7

About

We bring you the latest biotechnology news from best research centers and universities around the world. Check our website.

Follow us

Recent News

Custom-made prostheses provide functional shoulder reconstruction after scapular tumor resections

Subtle BRCA2 Gene Alterations Link to Moderate Breast Cancer Risks

Degraded Tibetan Desert Steppes Show Leaf Carbon Isotope Enrichment Under Water Limitation

Subscribe to Blog via Email

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Join 88 other subscribers
  • Contact Us

Bioengineer.org © Copyright 2023 All Rights Reserved.

Welcome Back!

Login to your account below

Forgotten Password?

Retrieve your password

Please enter your username or email address to reset your password.

Log In
No Result
View All Result
  • Homepages
    • Home Page 1
    • Home Page 2
  • News
  • National
  • Business
  • Health
  • Lifestyle
  • Science

Bioengineer.org © Copyright 2023 All Rights Reserved.