AI chatbots offer real empathy for chronic illness distress but stumble when symptoms turn serious

Millions of people living with chronic conditions now turn to general-purpose AI chatbots as de facto first-contact environments for psychological distress, asking whether their reactions are normal, seeking coping strategies, or deciding whether professional help is warranted. A new standardized simulated-patient benchmarking study published in Frontiers in Psychiatry tested whether three widely accessible conversational AI systems could deliver psychological support that is not merely empathic, but clinically proportionate, sensitive to the chronic illness context, and balanced in its interpretation of psychological and somatic factors. The findings reveal a striking gap between emotional recognition, which appears robust across leading systems, and the ability to interpret distress within the lived structure of chronic illness, which remains substantially less mature.

The research team, led by investigators from Guilin Medical University, Xiangnan University, and The Chinese University of Hong Kong, compared ChatGPT, Gemini, and Doubao across twenty simulated-patient scenarios spanning two contrasting chronic conditions: type 2 diabetes mellitus (T2DM) and chronic spontaneous urticaria (CSU). Each scenario was independently submitted to all three systems in two testing rounds, yielding 120 responses. Blinded raters with medical degrees assessed overall psychological support appropriateness, recognition and emotional support, chronic-illness contextualization, psychological support and referral, psychosomatic balance and safety, and major clinical or safety errors. Repeated testing assessed response consistency.

Overall psychological support appropriateness differed significantly across systems (P = 0.002), with 90.0% of ChatGPT, 75.0% of Gemini, and 55.0% of Doubao responses meeting the prespecified appropriateness threshold. Recognition and emotional support was consistently strong across all three systems, with no significant difference (P = 0.247). However, greater differences emerged in chronic-illness contextualization (P = 0.001), psychological support and referral (P = 0.002), and psychosomatic balance and safety (P < 0.001). The clearest and most consistent separation was between ChatGPT and Doubao, with ChatGPT retaining higher scores after Holm adjustment for contextualization (adjusted P = 0.008), psychological support and referral (adjusted P = 0.014), and psychosomatic balance and safety (adjusted P = 0.005).

The most consequential finding concerned symptom severity. All 24 responses to mild presentations were classified as psychologically appropriate, compared with only 14 of 24 responses to moderate presentations (exact permutation P = 0.016). This indicates that performance became substantially less reliable when digital support required stronger escalation toward professional assessment. Recognition and emotional support remained stable across severity levels (P = 0.500), suggesting that the systems could still identify distress even when they failed to recalibrate contextual interpretation, proportional referral, and psychosomatic balance as clinical severity increased.

Contextualization was also less consistent in CSU than in T2DM. Among the 48 responses generated from matched depression and anxiety scenarios, 20 of 24 T2DM responses (83.3%) and 18 of 24 CSU responses (75.0%) were classified as psychologically appropriate, with no significant difference in overall appropriateness between the two contexts (P = 0.313). However, chronic-illness contextualization was significantly lower in CSU (P = 0.008), particularly when responses needed to incorporate flare unpredictability, pruritus, sleep disruption, and illness uncertainty. The condition-specific scenarios reinforced this distinction: five of six responses to the two T2DM distress variants were classified as appropriate, compared with only one of six responses to the two CSU flare- and uncertainty-related distress variants.

Major clinical or safety errors occurred in 0%, 10.0%, and 15.0% of first-round ChatGPT, Gemini, and Doubao responses, respectively. No system generated a major unsafe medication or treatment recommendation. The errors that did occur were principally failures of calibration rather than overtly hazardous treatment advice: three involved disproportionate psychological attribution of physical or illness-related symptoms, and two reflected insufficient escalation in moderate presentations with meaningful functional impairment. Inter-rater agreement was substantial across all evaluated domains, with linearly weighted Cohen’s κ ranging from 0.79 to 0.88 for the secondary domains and 0.84 for overall appropriateness.

Test–retest consistency was moderate and varied between systems. Exact agreement in overall appropriateness was 85.0% for ChatGPT, 75.0% for Gemini, and 65.0% for Doubao, with corresponding linearly weighted κ values of 0.69, 0.59, and 0.49. Agreement in the binary psychologically appropriate classification was higher than exact ordinal agreement, indicating that some repeat variability reflected movement within rather than across the clinically relevant appropriateness threshold. This graded reduction in response-level stability across systems suggests that even the best-performing model cannot yet guarantee reproducible clinical formulation across identical prompts.

The study introduces the concept of psychosomatic calibration as a more useful framework for evaluating AI in chronic illness than diagnostic accuracy alone. Psychosomatic calibration refers to the capacity to preserve the clinical relationship between bodily symptoms, psychological distress, and disease context without collapsing one into the other. Over-psychologising a physical symptom can be as clinically misleading as ignoring the psychological burden of a chronic condition. Conversely, repeatedly attributing distress to the disease itself can obscure the emergence of a clinically significant depressive or anxiety syndrome. The challenge is not to choose between a psychological and somatic explanation, but to recognize when both remain plausible and when further assessment is warranted.

The contrast between T2DM and CSU illustrates why this matters. T2DM imposes a relatively structured psychological burden shaped by self-monitoring, treatment adherence, dietary regulation, and repeated evaluation of disease control, experiences that map readily onto established narratives of treatment fatigue and diabetes distress. CSU poses a different interpretive problem, shaped by recurrence, unpredictability, pruritus, disrupted sleep, and uncertainty about triggers. Such experiences do not map as neatly onto standard psychological templates. A response that merely identifies anxiety may therefore be technically plausible yet clinically shallow, because what is required is not simply recognition of affect but an understanding of the structure of uncertainty that sustains it.

The broader conclusion is not that general-purpose AI succeeds or fails at psychological support in chronic illness, but that the frontier has moved. Basic emotional recognition is no longer the most informative test of model capability. The harder problem is whether AI can formulate distress in context, preserve appropriate uncertainty between psychological and somatic explanations, and adjust its recommendations as clinical severity changes. The authors suggest that implementation should emphasize context-sensitive guidance, severity-responsive referral, and clear integration with rather than substitution for professional care. As AI becomes increasingly embedded in how patients understand and respond to chronic illness, evaluation should shift from asking whether models can produce plausible mental-health responses to asking whether those responses are appropriately contextualized, proportionate, and reproducible.

Subject of Research: AI chatbots

Article Title: Psychological support for depression and anxiety across chronic illness contexts: a comparative evaluation of three general-purpose AI chatbots

Article References: Xiao, W., Cao, X., Jin, W., Chen, X., Cai, J., Luo, X., & Deng, J. (2026). Psychological support for depression and anxiety across chronic illness contexts: a comparative evaluation of three general-purpose AI chatbots. Frontiers in Psychiatry, 17, Article 1962615. https://doi.org/10.3389/fpsyt.2026.1962615

Image Credits: AI Generated

DOI: 10.3389/fpsyt.2026.1962615

Keywords: AI chatbots, ChatGPT, Gemini, Doubao, depression, anxiety, chronic illness, type 2 diabetes, chronic spontaneous urticaria, digital mental health, psychosomatic medicine, large language models