Language Models Accurately Extract Urodynamic Data but Show Mixed Diagnostic Agreement

In a blinded comparative study evaluating artificial intelligence as an aid in urological care, contemporary large language models demonstrated flawless accuracy in extracting primary numerical measurements from invasive urodynamic reports while achieving substantial diagnostic agreement with medical specialists for specific conditions such as detrusor underactivity. The retrospective investigation, published in BMC Urology by researchers Hüseyin Koçakgöl and Muhittin Atar, directly examined how advanced generative models perform when tasked with interpreting complex clinical urodynamic records alongside human experts. Although artificial intelligence systems are increasingly being explored across various medical disciplines as prospective clinical decision-support tools, their practical utility and diagnostic reliability within the technical domain of invasive urodynamic interpretation had previously remained poorly characterized.

To establish an objective benchmark of diagnostic performance, the research team conducted a retrospective analysis centered on a cohort of 113 urodynamic studies, comprising records from 62 female patients and 51 male patients. Each patient report was anonymized and compiled in standard portable document format before undergoing evaluation. The study design incorporated two experienced urologists who served as the expert reference standard, independently and blindly assessing every anonymized record without access to peer determinations or algorithmic findings. In parallel, the exact same collection of urodynamic reports was submitted to two prominent artificial intelligence platforms, specifically GPT-4o and Claude Sonnet 4, allowing investigators to observe how state-of-the-art foundation models handle real-world clinical records under rigorous testing conditions.

The researchers structured the automated evaluation around two distinct prompting methodologies to probe how differing instructional styles shape model performance. The first technique employed rule-based prompting, which supplied the systems with predefined diagnostic criteria directly incorporating International Continence Society recommendations along with explicit operational thresholds. The second method utilized intuitive prompting, directing the models to emulate holistic clinical reasoning analogous to the interpretive approach applied by practicing physicians. By applying both prompting paradigms across both GPT-4o and Claude Sonnet 4, the investigators established four independent computational analysis arms designed to evaluate diagnostic classifications across six specific clinical categories: bladder outlet obstruction, detrusor underactivity, hypocompliant bladder, hypercompliant bladder, atonic bladder, and normal urodynamic study.

To evaluate reliability and diagnostic concordance across all human and algorithmic assessments, the investigators applied standardized statistical metrics, measuring inter-rater agreement through Cohen’s kappa statistics and evaluating numerical data alignment through Pearson correlation coefficients. Prior to assessing machine accuracy, the study established the baseline level of consistency achieved between the two participating clinicians. The interobserver agreement between the two experienced urologists spanned from substantial to almost perfect, yielding an overall mean Cohen’s kappa of 0.727 across all evaluated categories. Clinician concordance reached its absolute peak in the identification of detrusor underactivity, where the two specialists recorded a kappa value of 0.942, reflecting exceptional diagnostic alignment between experienced practitioners.

When evaluating the models’ capacity to retrieve quantitative information from the source reports, the investigators observed remarkable consistency across every automated configuration. All four artificial intelligence analysis arms achieved identical extraction of primary numerical urodynamic parameters, demonstrating a perfect Pearson correlation coefficient of r = 1.000 when compared against the extracted baseline data. However, this absolute consistency did not extend uniformly to secondary computational values. The researchers documented noticeable variability across the models when deriving secondary calculated metrics, specifically the bladder outlet obstruction index and the bladder contractility index, whenever the systems operated under intuitive prompting strategies that relied on unstructured clinical deduction rather than rigid calculation rules.

Diagnostic agreement between the language models and the urologists varied considerably depending on the specific medical condition and the prompting strategy applied. The highest degree of concordance between the models and human clinicians occurred in the diagnosis of detrusor underactivity and bladder outlet obstruction under the Claude-Intuitive configuration, which achieved Cohen’s kappa scores reaching up to 0.753 for detrusor underactivity and 0.522 for bladder outlet obstruction. Conversely, the rule-based prompting framework yielded its strongest performance in categories governed by strict numerical cutoffs, attaining the highest agreement metrics across several threshold-defined diagnoses where predefined parameters and operational thresholds directly guided the classification process.

A notable observation emerged when the research team isolated patient cases where both independent urologists were in complete agreement. Within these consensus subsets, diagnostic concordance between the language models and the specialists increased substantially. In the full-consensus cohort of 67 studies where both human clinicians reached identical diagnostic conclusions across all categories, the agreement for detrusor underactivity rose to a kappa of 0.802. Similarly, within a diagnosis-specific consensus group comprising 110 cases, the agreement metric for detrusor underactivity reached a kappa of 0.769. These findings indicated that a substantial portion of the measured disagreement between artificial intelligence and human clinicians was concentrated within ambiguous cases that lacked concordant evaluations between the expert urologists themselves.

Despite the high level of precision shown in numerical data retrieval and select diagnostic categories, the authors emphasized the inherent boundaries of their findings and the defined scope of current generative models. Because the investigation evaluated six predefined diagnostic categories under two specific prompt styles, the results illustrate targeted capabilities rather than comprehensive urodynamic interpretive competence or diagnostic autonomy. The researchers concluded that contemporary large language configurations demonstrate strong potential as adjunctive decision-support tools for specific urodynamic parameters, particularly when assisting with well-defined conditions, but should not be viewed as substitutes for independent clinical judgment or qualified expert evaluation.

Subject of Research: Comparison of large language models and urologists in invasive urodynamic interpretation

Article Title: Large language models for invasive urodynamic interpretation: a blinded comparison with experienced urologists

Article References: Koçakgöl, H., & Atar, M. (2026). Large language models for invasive urodynamic interpretation: a blinded comparison with experienced urologists. BMC Urology. https://doi.org/10.1186/s12894-026-02401-0

Image Credits: AI Generated

DOI: 10.1186/s12894-026-02401-0

Keywords: invasive urodynamic interpretation, urodynamics, large language models, artificial intelligence in urology, GPT-4o, Claude Sonnet 4, detrusor underactivity, bladder outlet obstruction, hypocompliant bladder, hypercompliant bladder, atonic bladder, normal urodynamic study, rule-based prompting, intuitive prompting, Cohen's kappa, Pearson correlation, clinical decision support, bladder contractility index