Study Evaluates AI Models on Simulated Pediatric Paracetamol Poisoning Cases

A comparative scenario-based study evaluated how three contemporary large language models handle simulated pediatric paracetamol poisoning cases. Researchers presented thirty hypothetical poisoning scenarios to ChatGPT 5.5, Claude Sonnet 4.6, and Gemini 3 Flash using public browser interfaces and a standardized single-turn prompt, collecting a total of ninety clinical responses. The investigation assessed whether advanced artificial intelligence systems could generate safe and guideline-concordant guidance for managing paracetamol overdoses in children, which represent a frequent reason for pediatric poisoning consultations and emergency department admissions.

A three-clinician expert panel evaluated the generated answers while blinded to model identity. Scoring was determined by consensus against a predefined reference management algorithm derived primarily from Australia and New Zealand clinical toxicology guidelines. The clinician panel applied multiple standardized measurement instruments to evaluate each response, including a study-specific Artificial Intelligence Evaluation Score for Clinical Poisoning Vignettes with a score range from 5 to 25, an eight-item key-action checklist scored from 0 to 8, and the Global Quality Scale ranging from 1 to 5.

The evaluation framework also tracked reference-standard concordance and documented the presence of any critical safety errors within each model output. Statistical comparisons across the three artificial intelligence platforms were carried out using the Friedman test, supplemented by Wilcoxon signed-rank tests with a Bonferroni correction for pairwise assessments. Differences in critical safety error occurrences were evaluated using the Cochran Q test.

Across the simulated cases, Claude Sonnet 4.6 attained higher performance marks than both ChatGPT 5.5 and Gemini 3 Flash, with statistically significant advantages across the primary assessment metrics according to Friedman test results. Pairwise comparisons indicated that ChatGPT 5.5 and Gemini 3 Flash demonstrated similar overall evaluation scores without statistically significant performance differences between them.

Safety evaluations demonstrated notable variations in clinical reliability across the tested models. Claude Sonnet 4.6 maintained full concordance with the clinical reference standard across all thirty presented scenarios and generated zero critical safety errors. In contrast, ChatGPT 5.5 produced one critical safety error across its thirty responses, while Gemini 3 Flash generated three critical safety errors among its thirty responses.

Although absolute numbers of critical errors varied across the three platforms, the overall difference in critical safety error rates among the systems did not reach statistical significance under the Cochran Q test, which yielded a value of 4.67 and a corresponding p-value of 0.097. The presence of inaccurate recommendations in simulated high-stakes emergency workflows underscored ongoing safety concerns regarding automated advice generation.

The study authors concluded that while contemporary large language models demonstrate potential as adjunctive decision-support instruments in pediatric toxicology, model discrepancies and the hazard of confident errors remain evident. Given these findings, researchers emphasized that artificial intelligence outputs in clinical toxicology must remain subject to direct expert verification and professional clinical supervision rather than autonomous implementation.

The research, authored by Alper Kaçar, Belen Ateş, and Sevilay Kök and published in BMC Pediatrics, addresses an area of clinical decision support where published evidence has remained scarce. Although large language models are increasingly examined for clinical guidance roles, their specific reliability in pediatric toxicology presents unique challenges because paracetamol overdose represents a frequent cause for emergency visits and specialized poisoning consultations in children.

To rigorously test the models against clinical benchmarks, the cross-sectional investigation compared three specific platforms: ChatGPT 5.5, Claude Sonnet 4.6, and Gemini 3 Flash. Evaluators queried each tool through public browser interfaces rather than customized backend environments, utilizing standardized single-turn prompts to simulate immediate real-world clinician interactions. Responses were judged against a reference standard derived primarily from Australia and New Zealand clinical guidelines, focusing on guideline-based accuracy, key-action completion, and overall response quality.

In their conclusions, the investigators observed that Claude Sonnet 4.6 demonstrated the highest overall quality, highest key-action execution, and superior guideline-based accuracy among the tested tools. Across all simulated pediatric scenarios, it produced no critical safety lapses and stayed completely concordant with the reference standard, whereas ChatGPT 5.5 generated one critical error and Gemini 3 Flash made three.

Despite the strong performance displayed by Claude Sonnet 4.6, the researchers stressed that contemporary models should only be considered as adjunctive decision-support aids rather than standalone systems. Because inter-model safety variations persist and large language models pose an inherent risk of delivering confident yet erroneous guidance in high-stakes clinical scenarios, any real-world use in pediatric poisoning management requires clinical verification and direct expert supervision.

Subject of Research: Comparative evaluation of three large language models managing simulated pediatric paracetamol poisoning

Article Title: Comparative performance and safety of three large language models in pediatric paracetamol poisoning management

Article References: Kaçar, A., Ateş, B., & Kök, S. (2026). Comparative performance and safety of three large language models in pediatric paracetamol poisoning management. BMC Pediatrics. https://doi.org/10.1186/s12887-026-07949-2

Image Credits: AI Generated

DOI: 10.1186/s12887-026-07949-2

Keywords: pediatric paracetamol poisoning, large language models, clinical decision support, ChatGPT 5.5, Claude Sonnet 4.6, Gemini 3 Flash, pediatric toxicology, emergency department visits, guideline concordance, critical safety errors, AIES-CPV, Global Quality Scale, scenario-based study, clinical vignettes, expert clinician panel, toxicology guidelines