ChatGPT, Grok and Gemini tested on prenatal spina bifida counseling questions

Three widely used artificial intelligence chatbots have been evaluated on how they answer parental questions after a prenatal diagnosis of spina bifida. In a new study published in Archives of Gynecology and Obstetrics, researchers submitted twenty frequently asked parental questions about a prenatal spina bifida diagnosis to ChatGPT, Grok, and Gemini, and then asked specialist physicians to grade the answers for scientific accuracy, quality, and readability.

The study was designed as a scenario-based evaluation.

The evaluation relied on established instruments rather than ad hoc impressions. The reviewers applied the JAMA Benchmark Criteria, the Global Quality Score, and the Modified DISCERN Scale, and also rated the responses on a 5-point Likert scale. Readability was assessed using the Ateşman Readability Formula. To check how consistently the specialists judged the answers, the researchers calculated interrater association with Kendall’s Tau-b and absolute agreement with intraclass correlation coefficients.

The headline result was broadly favorable to the chatbots. All three AI tools received high expert-rated scientific accuracy scores and high overall evaluation scores, and every model achieved full scores on the JAMA criteria.

There were nonetheless meaningful differences among the models. The researchers found significant overall differences among the three AI tools for the Global Quality Score, the DISCERN score, and the Likert ratings, using the Friedman test with all p values below 0.05. After Bonferroni-corrected pairwise comparisons using the Wilcoxon signed-rank test with exact p values, ChatGPT scored significantly higher than Grok on the DISCERN and Likert scales. No other pairwise comparison reached statistical significance after correction, including ChatGPT versus Gemini on any scale, so the ranking of the three models was not a clean hierarchy.

Readability emerged as a consistent strength. The readability scores for the chatbot responses corresponded to a high school reading level, suggesting that the answers were written at a level accessible to many lay readers rather than only to medical professionals.

The agreement between the expert raters was more complicated. Inter-rater association and absolute agreement varied across the three models: alignment was stronger for Gemini, estimates for Grok were lower and not statistically significant, and results for ChatGPT had limited interpretability because of insufficient score variability.

The authors conclude that AI tools can generate scientifically accurate and comprehensive prenatal-counseling information for spina bifida, as judged against a standard clinical reference, ACOG Practice Bulletin No. 187.

At the same time, the study’s conclusion is explicit about limits. The researchers state that these tools cannot replace physician–patient interaction, but they may serve as useful adjuncts in patient education and counseling. The finding is about the quality of text answers to fixed questions, not about the safety of AI in real clinical decision-making, and the evaluation involved a single scenario, twenty questions, one language, and a small panel of specialist raters.

For families navigating a prenatal spina bifida diagnosis, the study suggests that widely available chatbots can now produce information that specialists judge accurate, high quality, and readable, at least when measured against a recognized clinical standard. The authors position such tools as supplements to, not substitutes for, the conversation between parents and their physicians, in which individual circumstances, emotions, and options for care are discussed in person.

Subject of Research: Evaluation of AI chatbot responses to parental questions about prenatal spina bifida counseling

Article Title: Artificial intelligence in the physician’s seat: a scenario-based evaluation of scientific accuracy, quality, and readability

Article References: Akbas, H., & Ozgun, N. (2026). Artificial intelligence in the physician’s seat: a scenario-based evaluation of scientific accuracy, quality, and readability. Archives of Gynecology and Obstetrics. https://doi.org/10.1007/s00404-026-08594-x

Image Credits: AI Generated

DOI: 10.1007/s00404-026-08594-x

Keywords: artificial intelligence, ChatGPT, Grok, Gemini, spina bifida, prenatal counseling, patient education, health information quality, perinatology, pediatric neurology, readability, DISCERN