Large language models evaluated on patient inquiries regarding asthma during pregnancy demonstrated uneven clinical quality and notable safety risks, with nearly one-third of generated answers containing safety concerns despite high marks for certain systems. In a cross-sectional comparative study conducted by investigators from Charité University Medicine Berlin and Zhejiang Hospital, researchers assessed how current artificial intelligence platforms handle high-risk reproductive health topics. The findings revealed that even when artificial intelligence applications produce answers judged favorably by clinical raters, their outputs frequently introduce clinical hazards or communication barriers that preclude unsupervised patient deployment.
Patients increasingly turn to artificial intelligence interfaces to interpret medical conditions and treatment choices, yet the suitability of these systems for sensitive reproductive medicine remains uncertain.
To establish a representative and standardized curriculum of patient-facing inquiries, the research team implemented a two-stage modified Delphi-informed consensus process. The selection procedure began with an initial pool of 40 candidate questions reflecting common dilemmas and management challenges surrounding asthma during pregnancy. Through iterative expert review, the evaluators narrowed the pool to 30 candidates before finalizing 19 retained questions specifically structured to test the breadth and depth of model knowledge across typical patient scenarios.
The experimental protocol tested six distinct large language models, prompting each system once per retained question under a standardized structured prompt. This testing matrix generated 114 distinct model responses and 342 rater-level evaluation records for detailed comparative analysis.
Three clinical evaluators independently examined the resulting outputs across several validated assessment instruments. The evaluators scored overall clinical response quality alongside information quality using the standardized DISCERN instrument, which evaluates health information reliability and treatment descriptions. In addition, the raters scored understandability and actionability using the Patient Education Materials Assessment Tool, known as PEMAT, while computational analyses calculated four established readability indices to determine the structural complexity of the generated text.
Statistical evaluation using matched question-level nonparametric tests with multiplicity correction and effect sizes revealed significant divergence in clinician-rated overall clinical response quality across the six evaluated architectures. The Friedman test indicated substantial variation among the systems (Friedman chi-square [5, N = 19] = 72.31, P < 0.001; Kendall’s W = 0.76). While GPT-5.5 and Claude Sonnet 5 achieved the highest observed median quality scores among the cohort, statistical testing demonstrated no significant difference between the two top-performing models after applying multiplicity corrections.
Beyond overall clinical response scores, performance across secondary metrics proved highly fragmented among the competing systems. Model rankings fluctuated substantially across DISCERN information quality, PEMAT understandability, PEMAT actionability, and automated readability calculations, with no single model demonstrating consistent leadership across all evaluated outcome measures. Crucially, none of the six evaluated language models achieved a median Flesch-Kincaid grade level at or below grade 8, indicating that all tested systems generated text too complex for general patient education standards.
Patient safety emerged as a primary concern during the investigation’s dedicated revision-stage safety analysis. Two clinicians independently reviewed all 114 responses using a predefined, structured safety taxonomy, demonstrating an inter-rater agreement of 87.7 percent for the presence of any safety concern (Cohen’s kappa = 0.70). Any response marked by rater disagreement or prespecified severity triggers underwent blinded adjudication by a third senior clinician, who ultimately confirmed that 37 of the 114 responses (32.5 percent) contained at least one safety concern, including 4 responses (3.5 percent) classified as high-risk unsafe advice.
The study authors concluded that while large language models show promising capabilities under structured prompting conditions, their performance remains fundamentally uneven across maternal asthma inquiries. High clinician-rated overall clinical response quality did not guarantee superior information quality, usability, appropriate reading levels, or clinical safety. Because the findings fail to demonstrate that current artificial intelligence models can be safely utilized without supervision, the authors emphasized that all model-generated patient guidance must undergo guideline-based clinical verification, explicit safety adjudication, and professional readability editing before clinical implementation.
Subject of Research: Multidimensional evaluation of large language models for patient education on asthma during pregnancy
Article Title: Multidimensional evaluation of large language models for patient education on asthma during pregnancy: a cross-sectional comparative study
Article References: Zhu, Z., Xie, C., & Wu, H. (2026). Multidimensional evaluation of large language models for patient education on asthma during pregnancy: a cross-sectional comparative study. Frontiers in Digital Health, 8, Article 1958383. https://doi.org/10.3389/fdgth.2026.1958383
Image Credits: AI Generated
DOI: 10.3389/fdgth.2026.1958383
Keywords: large language models, asthma during pregnancy, patient education, reproductive medicine, artificial intelligence in medicine, digital health, clinical response quality, DISCERN, PEMAT, readability indices, Flesch-Kincaid grade level, patient safety, clinical adjudication, cross-sectional study, Delphi method, health informatics, maternal health, medical advice safety
News Source: Harold Sullivan. (November 11, 2026). AI Models Show Inconsistent Performance and Safety Risks in Pregnancy Asthma Advice. Scienmag.



