Professional healthcare providers and heart failure patients hold significantly diverging perspectives on the quality of self-management education content generated by artificial intelligence, according to a cross-sectional study evaluating five prominent large language models. Healthcare professionals conducting clinical evaluations awarded their highest score to DeepSeek, whereas heart failure patients rated Doubao as their top choice, followed by DeepSeek. The findings demonstrate a notable divide between professional clinical benchmarks and patient reception, revealing that the artificial intelligence systems that clinicians determine to be the most accurate and safe do not always correspond to the tools that patients perceive as the most comprehensible, useful, and trustworthy.
Published in BMC Health Services Research, the cross-sectional investigation addressed a well-recognized barrier in cardiology and chronic disease care: while patient self-management is essential for improving clinical outcomes in heart failure, adherence to self-care regimens remains suboptimal, particularly among older adults and individuals with limited health literacy. Large language models possess the technical capability to generate educational materials at scale, but empirical evidence evaluating their clinical accuracy, safety, and alignment with the specific needs of heart failure patients has remained limited. To investigate this question, researchers designed a study to compare the quality of responses generated by five widely used models—DeepSeek, Doubao, Grok, Gemini, and ChatGPT—and to quantify the discrepancies between clinician evaluations and patient perceptions.
The research protocol relied on a structured 45-item question bank focused on self-management among heart failure patients. This assessment instrument encompassed six distinct clinical domains and incorporated items varying in both question format and difficulty level to test the models across a diverse spectrum of patient care scenarios. All 45 questions were administered to the five large language models under standardized conditions. The resulting model-generated answers were then compiled for independent, dual-perspective evaluation by clinical specialists and by patients living with heart failure.
Professional evaluation was carried out independently by four healthcare professionals who examined the responses across four critical domains: accuracy, completeness, safety, and actionability. The clinicians graded each response using a 1 to 5 Likert scale that incorporated an explicit safety veto mechanism designed to flag and penalize medically hazardous content. In these clinical assessments, DeepSeek achieved the highest overall score among the evaluated systems, recording a score of 19.49 out of a maximum of 20 points. Statistical evaluation utilizing generalized estimating equations established that DeepSeek scored significantly higher than the other four models, with beta coefficients ranging from −1.79 to −1.06 and all comparisons reaching statistical significance at P < 0.001.
In contrast to the clinician rankings, an evaluation conducted by fifteen heart failure patients yielded a different ordering of model preferences. The patient evaluators appraised the generated content across four distinct dimensions: comprehensibility, usefulness, satisfaction, and trustworthiness. Among the fifteen patients, Doubao achieved the highest overall score at 16.93, followed by DeepSeek with a score of 16.33. This preference for Doubao was especially evident among demographic subgroups, including older patients and individuals with lower levels of education, who favored the model for its readability and perceived trustworthiness.
The magnitude of the discrepancy between clinical evaluation scores and patient perception ratings varied substantially across the five systems, ranging from a low divergence of 1.50 for Doubao to a wide gap of 5.02 for ChatGPT. The researchers utilized generalized estimating equations, intraclass correlation coefficients, and linear mixed-effects models to compare performance and assess variability across raters and models. These statistical comparisons highlighted the extent to which patient ratings can diverge from clinical benchmarks depending on the specific model evaluated.
The study identified substantial safety vulnerabilities in specific clinical areas, most notably concerning dietary self-care. Across all five models, responses addressing dietary sodium restriction generated consistent safety concerns during professional evaluation, with diet and sodium restriction emerging as the weakest-performing domain in the entire study. Strikingly, despite the identified safety flaws and the clinical safety veto criteria applied by the healthcare professionals, patient participants consistently rated these same dietary responses favorably. The authors concluded that professional and patient evaluations of large language model content diverge significantly, emphasizing that clinical accuracy alone is insufficient and underscoring the urgent necessity of balancing strict clinical quality with patient-centered design in healthcare artificial intelligence.
Subject of Research: Evaluation of large language models for heart failure patient self-management education
Article Title: Comparative analysis of five large language models for generating heart failure patient education content: dual perspectives from professionals and patients
Article References: Chen, Y., Peng, Y., Chen, S., Huang, B., Xie, J., Xue, Y., Wu, Q., Gao, W., Chen, L., & Lin, Y. (2026). Comparative analysis of five large language models for generating heart failure patient education content: dual perspectives from professionals and patients. BMC Health Services Research. https://doi.org/10.1186/s12913-026-15860-w
Image Credits: AI Generated
DOI: 10.1186/s12913-026-15860-w
Keywords: heart failure, large language models, patient education, self-management, artificial intelligence in healthcare, DeepSeek, Doubao, health literacy, Comparative, analysis, five, large
News Source: Courtney Benton. (November 10, 2026). Clinicians and Patients Evaluate Five AI Models for Heart Failure Education. Scienmag.



