Chatbots answering questions about male infertility changed markedly between 2025 and 2026, but not all of them changed for the better, according to a repeated evaluation published in Basic and Clinical Andrology.
The central finding was that performance evolved in a model-specific rather than uniformly progressive manner. ChatGPT improved significantly across all four scored domains after its underlying model was updated from ChatGPT-4o to ChatGPT-5.5. DeepSeek, which had been the weakest performer in 2025, showed the most pronounced year-on-year improvement, particularly in reliability and expert opinion scores. Perplexity, by contrast, became easier to read but lost ground on global quality and expert-rated clinical usefulness, despite remaining the platform most consistently associated with citations in the earlier assessment.
The study was designed as a repeated cross-sectional comparison. The authors chose male infertility because it is a guideline-sensitive field in which inaccurate or incomplete information can affect diagnostic evaluation, counseling and treatment expectations. Because the relevant male infertility recommendations remained largely stable between the 2025 and 2026 EAU Sexual and Reproductive Health Guidelines, any change in chatbot performance could be attributed more plausibly to changes in model behavior, retrieval strategy and source selection than to shifts in the underlying clinical reference standard.
The question set was built by directly transforming recommendation statements from the EAU guideline section into concise, neutral, information-seeking prompts. All prompts were written in formal English without leading phrasing, contextual cues or follow-up questions. Each of the 88 prompts was submitted separately to each platform in a fresh private chat session with memory disabled, and only the first complete response was retained for analysis. The authors state that this protocol was chosen to reflect a real-world first-query scenario, a common pattern among patients and clinicians seeking rapid medical information.
Two academic urologists with expertise in male infertility scored every response independently, blinded to model identity and assessment year, with final scores averaged between reviewers. Reliability was measured with the modified short-form DISCERN instrument, a five-item binary scale covering clarity of aims, use of reliable sources, balance, provision of additional references and acknowledgment of uncertainty. Overall information quality was rated on the five-point Global Quality Score, readability was estimated with the Gunning Fog Index, which approximates the years of education needed to understand a text on first reading, and each answer also received an expert rating of clinical usefulness on a five-point scale. The intraclass correlation coefficient between the two reviewers was 0.87, indicating strong agreement.
In the 2025 assessment, significant inter-model differences emerged across all four domains. Perplexity achieved the highest median DISCERN, Global Quality Score and expert opinion scores, and also had the highest Gunning Fog Index, meaning its answers were the most linguistically complex. DeepSeek recorded the lowest median DISCERN and expert opinion values. Post hoc testing showed Perplexity differing significantly from both ChatGPT and DeepSeek on the content-related measures, while ChatGPT and DeepSeek did not differ significantly on readability.
A year later the picture had shifted considerably. DISCERN scores converged, with all three models reaching a median of 5 and overlapping interquartile ranges, so that inter-model differences on this reliability measure were no longer statistically significant. Significant differences persisted for readability, Global Quality Score and expert opinion. ChatGPT achieved the highest median Global Quality Score in 2026, while expert opinion favored ChatGPT and DeepSeek over Perplexity. Perplexity continued to produce the most linguistically complex responses, although its readability gap relative to ChatGPT narrowed enough that the difference between the two was no longer statistically significant.
The within-model comparisons quantified these trajectories. ChatGPT’s Gunning Fog Index fell from 15.68 to 14.99, and its DISCERN, Global Quality Score and expert opinion scores each rose by one median point, with all four changes statistically significant. Perplexity’s readability improved from a Gunning Fog Index of 17.39 to 16.00, but its Global Quality Score and expert opinion scores each dropped by one median point, and its DISCERN interquartile range widened. DeepSeek improved on every measure: its Gunning Fog Index fell from 15.30 to 14.30, DISCERN rose from a median of 3 to 5, expert opinion rose from 3 to 5, and its Global Quality Score remained at a median of 4 with an upward shift in distribution.
The safety evaluation added a cautionary note. Potential patient-safety concerns, defined as inaccurate, incomplete or insufficiently qualified information that could plausibly lead to inappropriate clinical interpretation, testing, treatment or delayed evaluation if acted upon without clinician verification, were identified in seven of the 88 DeepSeek responses from 2025, or 8.0 percent. No such concern was identified in any 2026 response. Importantly, none of the 528 responses contained an explicit diagnostic or therapeutic recommendation considered likely to cause immediate or serious patient harm. The authors emphasize that fluent, well-structured and convincing answers may still contain errors, omissions or inappropriate clinical prioritization, and that the absence of explicit high-risk advice should not be read as evidence that these platforms are suitable for unsupervised clinical use.
The authors also flag several limitations that shape how the results should be read. Only a single response was retained for each prompt-model-year combination, so the variability of outputs across repeated queries could not be assessed, and the findings represent sampled first-pass responses at specific time points rather than stable estimates of model performance or reproducibility. The comparator set excluded medical-specific or clinically grounded retrieval-augmented systems, so the results do not compare general-purpose platforms with specialized clinical tools. The questions, being directly derived from guideline statements, may have facilitated guideline-concordant answers and may not generalize to vague or colloquial user queries. The study was limited to English-language responses, user-facing platforms may undergo undisclosed backend changes, and expert scoring, although blinded, retains some subjectivity. The authors conclude that large language models may support rapid access to and summarization of male infertility information, but that verification against authoritative sources and appropriate human oversight remain necessary.
Subject of Research: Year-on-year changes in how ChatGPT, Perplexity and DeepSeek answer guideline-based male infertility questions
Article Title: Evolution of guideline-based male infertility responses by large language models between 2025 and 2026: a comparative analysis of ChatGPT, Perplexity, and DeepSeek
Article References: Elmaağaç, B., Özercan, A. Y., Demir, M., & Gül, M. (2026). Evolution of guideline-based male infertility responses by large language models between 2025 and 2026: a comparative analysis of ChatGPT, Perplexity, and DeepSeek. Basic and Clinical Andrology, 36(1), Article 30. https://doi.org/10.1186/s12610-026-00327-z
Image Credits: AI Generated
DOI: 10.1186/s12610-026-00327-z
Keywords: large language models, male infertility, ChatGPT, Perplexity, DeepSeek, EAU guidelines, DISCERN, Global Quality Score, readability, patient safety, artificial intelligence, andrology
News Source: Ophelia Keating. (October 28, 2026). ChatGPT, Perplexity and DeepSeek Show Diverging Shifts in Male Infertility Answers. Scienmag.
