Large language models such as ChatGPT have become an increasingly popular source of health information, offering patients immediate explanations of symptoms, treatments, and medical conditions. Yet a new evaluation of ChatGPT-4.0 suggests that confidence and fluency do not necessarily translate into clinically dependable advice. When tested against Canadian urological standards, the system produced answers judged appropriate in only 40% of cases, raising concerns about the risks of relying on artificial intelligence for independent medical decision-making.
The study, conducted by Wyatt MacNevin and colleagues at Dalhousie University, examined how accurately ChatGPT-4.0 answered common patient-oriented questions in urology. Rather than assessing the model against American or European recommendations, as many earlier investigations have done, the researchers used guidelines from the Canadian Urological Association, or CUA, as their benchmark. This distinction is important because recommendations can vary between professional organizations, particularly in areas involving diagnostic thresholds, treatment choices, screening practices, and the management of complex or borderline cases.
The researchers selected ten questions representing a broad cross-section of urological care. The topics included kidney stones, prostate cancer, benign prostatic hyperplasia, erectile dysfunction, overactive bladder, urinary tract infections, andrology, hypogonadism, pediatric urology, and kidney cancer. Each question was written in plain language to resemble the kind of request a patient might enter into a conversational artificial-intelligence system. The questions were submitted to the March 2025 version of ChatGPT-4.0 during three independent sessions, producing 30 responses for evaluation.
Three reviewers assessed the answers using a four-point Likert scale designed to distinguish between partial accuracy and clinically useful completeness. A score of zero indicated a completely incorrect answer, while a score of one represented a response containing both correct and incorrect information. A score of two meant that an answer was correct but inadequate, and a score of three indicated a comprehensive response. The investigators defined an appropriate answer as one scoring at least 2.00. This approach allowed the team to look beyond whether ChatGPT mentioned isolated facts and instead examine whether the overall response was sufficiently accurate and useful for a patient seeking reliable guidance.
Across all 30 responses, the model achieved a mean score of 1.64, with a standard deviation of 0.85. In practical terms, the average answer fell between “some correct and some incorrect” and “correct but inadequate.” Only 12 of the 30 responses met the study’s threshold for appropriateness. The findings indicate that a response can sound medically polished while still omitting essential context, presenting incomplete guidance, or including statements that do not fully align with Canadian recommendations. Such weaknesses are especially consequential in medicine, where a seemingly minor omission can influence whether a patient seeks urgent care, delays evaluation, or misunderstands the purpose of a treatment.
Performance varied according to the difficulty and subject of the question. Easy questions received a mean score of 1.87, compared with 1.31 for questions classified as medium difficulty, a difference reported as statistically significant at p < 0.05. The pattern suggests that ChatGPT performs more reliably when answering straightforward, fact-based questions with relatively definitive answers. More nuanced questions, by contrast, may require interpretation of symptoms, consideration of patient-specific risk factors, or careful comparison of competing recommendations—tasks that remain difficult for general-purpose language models.
The model performed particularly well in several domains. Questions involving prostate cancer, erectile dysfunction, andrology, and kidney cancer received perfect median scores of 3.00. The researchers suggest that these results may reflect the large volume of standardized and widely available online information related to these conditions. When a topic has consistent terminology, well-established treatment pathways, and abundant educational material, a language model may be more likely to generate a coherent and broadly accurate answer. However, a high score on a particular topic does not establish that the system is capable of diagnosis or individualized treatment planning.
Other areas proved more challenging. Responses concerning urinary tract infections, overactive bladder, nephrolithiasis, and hypogonadism received lower scores, even when some of the questions were considered easy. The researchers propose several possible explanations, including inconsistencies in publicly available medical content and differences between Canadian guidance and recommendations issued by other international organizations. Kidney stones and urinary infections, for example, can involve decisions that depend heavily on factors such as stone size and location, fever, obstruction, pregnancy, kidney function, antimicrobial resistance, or the presence of systemic illness. A generic answer may fail to communicate which symptoms require urgent assessment.
Despite its low overall appropriateness rate, ChatGPT showed substantial consistency across repeated questions. The mean variance was 0.27, suggesting that the model generally produced similar scores when the same questions were submitted independently. This consistency is technically meaningful but should not be confused with correctness. A system can reliably reproduce an incomplete or partially inaccurate answer. In other words, reproducibility may indicate stable model behavior, while offering no guarantee that the underlying medical content is aligned with current clinical practice.
The study arrives as patients increasingly use conversational artificial intelligence before speaking with a physician. ChatGPT can explain medical terminology, summarize general concepts, and help users prepare questions for a consultation. Its ability to generate fluent, personalized-sounding responses can also create an impression of authority, even though the system does not examine patients, verify their medical histories, interpret physical findings, or independently confirm every claim against the latest guidelines. The authors therefore caution that current outputs are not sufficient for unsupervised patient use and urge urologists to discuss the limitations of artificial-intelligence health tools proactively.
The researchers recommend that medical responses generated by large language models include mandatory disclaimers and that future systems incorporate authoritative clinical guidelines more directly during training or retrieval. Such integration could improve alignment with regional standards, although it would not eliminate the need for physician oversight. Prospective research will also be necessary to determine whether AI-generated advice changes patient behavior, affects access to care, or contributes to delayed diagnoses and inappropriate treatment. For now, the study’s central message is clear: ChatGPT may be a useful educational assistant, but its polished language should not be mistaken for clinical reliability. In general urology, the system produced appropriate responses in fewer than half of the tested cases, underscoring the need for rigorous validation before widespread adoption in patient care.
Subject of Research: Evaluation of ChatGPT-4.0’s accuracy and reliability in answering common urological questions using Canadian Urological Association guidelines.
Article Title: Assessing the utility of a natural language processing model in answering common urological questions
Article Publication Date: 20-Aug-2026
Web References: https://doi.org/10.1002/uro2.70028
References: Canadian Urological Association guidelines; MacNevin et al., “Assessing the utility of a natural language processing model in answering common urological questions,” UroPrecision.
Image Credits: Higher Education Press
Keywords: ChatGPT, artificial intelligence, large language models, urology, medical misinformation, Canadian Urological Association, patient health information, clinical guidelines, prostate cancer, kidney stones, urinary tract infections, healthcare technology
