AI Chatbots Write Safe Exercise Advice for Older Adults, but Readability Falls Short

Three widely used artificial intelligence chatbots can produce exercise awareness messages for older adults that experts judged accurate, safe, and free of fabricated information, according to a cross-sectional observational study published in the European Review of Aging and Physical Activity. Yet the same study found that every model wrote at a reading level far above what is recommended for health materials aimed at older people, leaving a clear gap between the quality of the content and its likely accessibility to the intended audience.

They compared ChatGPT-4o from OpenAI, Claude 3.5 Sonnet from Anthropic, and Deepseek-R1, querying all three systems via the web on July 15, 2025, in English. To avoid conversational memory bias, each condition was raised in a fresh dialogue session, and a single standardized prompt instructed each model to act as a health communication assistant specialized in geriatrics and exercise.

The prompt asked for a short awareness message of no more than 150 words, written in clear, simple language that a person aged 65 or older could easily understand. Each message had to address three explicit aspects: considerations for the exercise prescription, how the disease may express itself and progress and why exercise matters, and the recommended types of exercise. The message also had to end with one motivating sentence encouraging safe action, delivered as plain text without bullet points or references.

As the gold standard, the team used the section on the role of exercise in primary, secondary, and tertiary disease prevention from the 2021 global consensus on optimal exercise recommendations for healthy longevity in older adults, published by the International Conference on Frailty and Sarcopenia Research. Thirteen common chronic conditions of later life were selected: arthritis, cancer, chronic obstructive pulmonary disease, chronic kidney failure, heart failure, coronary artery disease, dementia, depression, osteoporosis, peripheral vascular disease, stroke, type 2 diabetes, and chronic venous insufficiency.

Two physicians, a geriatrician and a sports medicine physician, each with at least three years of clinical experience, independently rated every generated message while blinded to which model had produced it. They scored five dimensions on a five-point Likert scale: accuracy relative to the consensus recommendations, clarity for older adults, safety, behavioral relevance, and absence of fabrication. The authors note that the accuracy dimension reflects an expert judgment-based approximation of guideline adherence rather than a binary adherence score. Readability was measured separately using three standard indices: the Flesch-Kincaid Grade Level, the Gunning Fog Index, and the Simple Measure of Gobbledygook, all calculated through an online readability calculator.

The results placed Claude slightly ahead. The geriatrician gave Claude an overall mean score of 4.63 plus or minus 0.55 and the sports medicine physician 4.60 plus or minus 0.55, compared with 4.55 plus or minus 0.50 and 4.48 plus or minus 0.53 for ChatGPT-4, and 4.15 plus or minus 0.87 and 4.17 plus or minus 0.86 for Deepseek. Agreement between the two raters, measured with the intraclass correlation coefficient, was moderate at 0.668, with a 95 percent interval spanning from -0.074 to 0.790. Overall scores differed significantly across the three models on the Kruskal-Wallis test, and post-hoc comparisons with Bonferroni correction showed that both ChatGPT-4 and Claude scored significantly higher than Deepseek, while the difference between ChatGPT-4 and Claude was not statistically significant.

On individual dimensions, Claude achieved the highest accuracy, with a combined mean of 4.58 plus or minus 0.64, against 4.08 plus or minus 0.39 for ChatGPT-4 and 3.31 plus or minus 0.62 for Deepseek. ChatGPT-4 excelled in clarity with a perfect score of 5 plus or minus 0, which the authors attribute to a writing style favoring short sentences and accessible vocabulary. Strikingly, all three models earned perfect scores of 5 plus or minus 0 for absence of fabrication, meaning no invented or erroneous information was detected in any of the evaluated messages. Mean scores by condition ranged around 4.6 for osteoporosis and 4.5 for venous insufficiency and type 2 diabetes.

Readability told a different story. Median Flesch-Kincaid Grade Level values ranged from 10.84 for ChatGPT-4 to 10.93 for Claude, corresponding roughly to a 10th to 11th grade reading level. The Gunning Fog Index medians were even higher, between 11.66 and 12.49, and SMOG medians fell between 11.73 and 11.94. All of these values substantially exceed the 6th to 8th grade threshold generally recommended for patient-facing health materials, a meaningful limitation given how common low health literacy is among older adults. The authors caution, however, that these lexical indices capture only surface-level textual complexity and do not directly measure comprehension, cognitive load, or motivational impact. One statistically significant correlation emerged: Claude’s accuracy scores correlated with its FKGL values at r = 0.599, p = 0.030, which the authors interpret as suggesting that slightly denser text may preserve terminological precision and fidelity to guideline content.

The study has important limitations that the authors themselves emphasize. It covered only 13 conditions, limiting generalizability, and the moderate inter-rater agreement reflects the subjective nature of some assessed dimensions. Most critically, no older adults participated: the evaluation relied entirely on expert judgment and automated readability scores, which are surrogate measures that cannot confirm how well the intended recipients would actually understand or act on the messages. All messages were generated in English, restricting applicability to other linguistic and cultural contexts, and the standardized prompt may not represent the optimal use of each model. The findings also reflect only the specific model versions tested in July 2025, in a field where systems evolve rapidly.

The authors conclude that Claude, ChatGPT-4, and Deepseek show strong potential for generating accurate, safe, hallucination-free exercise awareness messages for older adults, but that readability must be optimized before clinical use, for example through prompts that explicitly demand shorter sentences and simpler vocabulary. They stress that these tools should complement, not replace, clinical expertise: physicians must review and adapt every message, and the absence of hallucinations in this study does not guarantee reliability across all contexts. End-user validation with older adults, incorporating comprehension testing and motivational assessment, remains an indispensable step before any clinical deployment, and randomized controlled trials will be needed to establish whether AI-generated messages actually improve adherence, physical activity, or health outcomes.

Subject of Research: Comparative evaluation of three large language models generating exercise awareness messages for older adults against the ICFSR 2021 consensus

Article Title: Quality of AI-generated exercise awareness messages for older adults aligned with the ICFSR consensus: a comparative study of three LLMs

Article References: Bayala, Y. L. T., Ouedraogo, L. S. P.-W., Cissé, A. R., Kaboré, F., Yameogo, A. R., Ouedraogo, A., Zongo, Y. E., Zabsonré Tiendrébeogo, W. J. S., & Ouedraogo, D.-D. (2026). Quality of AI-generated exercise awareness messages for older adults aligned with the ICFSR consensus: a comparative study of three LLMs. European Review of Aging and Physical Activity, 23(1), Article 38. https://doi.org/10.1186/s11556-026-00428-8

Image Credits: AI Generated

DOI: 10.1186/s11556-026-00428-8

Keywords: large language models, ChatGPT-4, Claude, Deepseek, exercise recommendations, older adults, ICFSR consensus, health communication, readability, patient education, geriatrics, artificial intelligence