In a prospective, blinded benchmarking study evaluating artificial intelligence for pediatric neurology patient education, linear mixed-effects models revealed significant performance variation across four large language models (F(3,27) = 8.76, p < 0.001).
Effective patient education is critical in pediatric neurology care, yet healthcare teams frequently face substantial consultation time constraints alongside variable health literacy among families. While large language models present a scalable solution for drafting educational materials, their factual accuracy and safety profiles within this highly specialized clinical domain have remained incompletely evaluated.
To rigorously assess model capabilities, investigators developed a prospective benchmarking protocol analyzing model responses to ten standardized pediatric neurology prompts across four platforms, generating 40 distinct scenario-model pairs. A blinded panel consisting of seven consultant pediatric neurology evaluators appraised the generated texts using an expert-developed rubric covering factual accuracy, safety, clinical appropriateness, and usability, relying on median scores across raters for all primary comparisons.
DeepSeek attained the highest overall composite performance score at 4.01 ± 0.61 while triggering zero safety flags, and it also secured the highest System Usability Scale rating at 74.2 out of 100. ChatGPT recorded a moderate overall composite score of 3.72 ± 0.75, and statistical analysis showed that the composite score difference between DeepSeek and ChatGPT was not statistically significant (p = 0.087).
Safety evaluations demonstrated notable hazards among the generated responses, with ChatGPT responsible for the majority of identified issues, accounting for five of the eight total safety flags recorded, which included a contraindicated medication recommendation. In comparison, Gemini and Perplexity achieved lower overall composite ratings, averaging 3.45 ± 0.77 and 3.38 ± 0.71, respectively.
The benchmark further demonstrated that the presence of even a single consensus safety flag dramatically undermined specialist confidence in the content. Across the evaluation cohort, the occurrence of any consensus safety flag reduced the odds of expert approval for caregiver-facing deployment after clinical review by more than ninety percent (odds ratio: 0.08, p < 0.001).
The authors stressed that the reported conclusions remain confined to the specific clinical prompts evaluated and do not establish generalized safety, clinical efficacy, real-world decision support, or direct caregiver comprehension, noting that no patient families or caregivers were directly examined. Consequently, the research team concluded that no current language model is equipped for autonomous healthcare communication, making mandatory clinician oversight an indispensable safeguard.
Methodologically, the prospective investigation relied on a structured evaluation architecture designed to test model behavior under controlled, blinded conditions. Seven consultant pediatric neurology specialists evaluated the responses generated by the four artificial intelligence platforms across ten standardized clinical prompts, resulting in forty distinct scenario-model pairs. Rather than relying on unstandardized impressions, the blinded raters utilized an expert-developed rubric specifically structured to measure four core parameters: factual accuracy, clinical safety, appropriateness, and perceived usability. Primary comparative calculations relied on median scores across the evaluating consultants, and linear mixed-effects modeling confirmed that performance differed significantly across the four systems (F(3,27) = 8.76, p < 0.001), isolating genuine system differences across the standardized prompt cohort.
The distribution of composite scores illuminated nuanced differences among the tested platforms, alongside important statistical caveats. DeepSeek achieved the highest numerical score across the panel at 4.01 ± 0.61, while ChatGPT recorded a composite rating of 3.72 ± 0.75. However, direct statistical testing showed that the overall composite difference separating DeepSeek and ChatGPT was not statistically significant (p = 0.087), indicating that composite metrics alone could not conclusively separate their aggregate quality. The remaining two platforms achieved lower overall composite scores, with Gemini registering 3.45 ± 0.77 and Perplexity recording 3.38 ± 0.71. In terms of user experience evaluated through the standardized System Usability Scale, DeepSeek attained a benchmark-leading score of 74.2 out of 100, reflecting favorable expert perceptions of its communicative presentation.
Across the entire prospective trial, raters recorded a total of eight consensus safety flags, five of which originated exclusively from ChatGPT responses. Most critically, these identified defects included an explicit recommendation for a contraindicated medication, underscoring the severity of errors that unverified systems can produce. By contrast, DeepSeek generated zero safety flags across the benchmark scenarios. The presence of any consensus safety flag proved catastrophic for expert endorsement: statistical modeling indicated that generating any consensus safety flag drastically reduced the odds of specialist approval for caregiver-facing use following clinical review, yielding an odds ratio of 0.08 (p < 0.001).
The investigators underscored strict boundaries regarding the generalizability of their benchmark, defining explicit limits to interpretation. The evaluation did not examine real-world clinical decision support, nor did it evaluate actual caregiver comprehension, as no patients or family caregivers participated in the assessment. Because findings remain strictly confined to the ten specific pediatric neurology prompts examined, the absence of safety flags in this scenario cohort does not establish broad clinical safety, therapeutic effectiveness, or confirmed caregiver benefit. The authors concluded that no contemporary large language model is suitable for autonomous clinical deployment, emphasizing that direct clinician verification remains an essential prerequisite while offering their prospective framework as a standardized methodology for specialty-specific artificial intelligence evaluation.
Subject of Research: Benchmarking large language models for pediatric neurology patient education
Article Title: Comparative benchmark of four selected large language models for generating patient education content in pediatric neurology: a prospective study
Article References: Muthaffar, O., Alyazidi, A., Awadalla, M., Alanezi, A. A., Alsehemi, M., Hothan, K., & Naseer, M. I. (2026). Comparative benchmark of four selected large language models for generating patient education content in pediatric neurology: a prospective study. Frontiers in Pediatrics, 14, 1901603. https://doi.org/10.3389/fped.2026.1901603
Image Credits: AI Generated
DOI: 10.3389/fped.2026.1901603
Keywords: pediatric neurology, large language models, patient education, DeepSeek, ChatGPT, Gemini, Perplexity, artificial intelligence, factual accuracy, medication safety, clinical appropriateness, System Usability Scale, linear mixed-effects models, clinician verification, prospective benchmarking, pediatric health literacy, safety flags
News Source: Harold Sullivan. (November 16, 2026). Pediatric Neurology Benchmark Finds Significant Performance Variation Across Four Language Models. Scienmag.



