Vignette Study Compares AI Models and Physicians in Tuberculosis Clinical Tasks

In an exploratory comparative investigation evaluating clinical decision-making across standardized tuberculosis scenarios, researchers observed no statistically significant differences in diagnostic accuracy or completeness between large language models and practicing physicians. The study, reported in the journal BMC Infectious Diseases, compared the clinical outputs of two prominent artificial intelligence systems—ChatGPT-4o and DeepSeek-R1—against two participating tuberculosis doctors. The comparative assessment evaluated how both groups addressed key stages of patient care, specifically focusing on tuberculosis triage, differential diagnosis, and management recommendations.

The research was initiated to explore whether large language models could assist in tuberculosis-related clinical tasks and to deliver preliminary comparative evidence on accuracy, completeness, and perceived output quality relative to experienced clinicians. Tuberculosis continues to stand as a leading cause of mortality worldwide, with clinical burdens falling heavily on low- and middle-income countries. In such healthcare settings, limited access to specialized tuberculosis physicians combined with fragmented patient information can contribute to delayed, variable, or inconsistent medical choices. Although computational language models have shown promise as clinical decision-support instruments, verified empirical evidence regarding their utility and dependability in tuberculosis care has remained scarce.

To establish a rigorous experimental baseline, the investigators constructed 17 standardized clinical case vignettes, comprising seven simulated patient scenarios and ten drawn directly from real-world clinical cases. Methodological reporting for the investigation was informed by STARD-AI guidance. Under the study protocol, both participating physicians and the two language models, ChatGPT-4o and DeepSeek-R1, generated their responses following standardized prompts and predetermined output formats. Expert-developed reference answers were established to serve as the definitive reference standard. To compare the resulting data and account for task structures, group-level differences between physicians and models were evaluated through linear mixed-effects models.

The primary outcomes of the investigation targeted response accuracy and completeness, quantified using standard information retrieval metrics: precision, recall, and the composite F1 score. Across the assigned clinical tasks, linear mixed-effects modeling showed no statistically significant performance gap between the artificial intelligence tools and the human physicians. In precision, the language models recorded an average score of 0.67 compared with 0.50 for the medical doctors, a difference that did not reach statistical significance (P = .642). In recall, the models achieved 0.53 while the clinicians reached 0.33 (P = .214). Similarly, the overall F1 score showed comparable performance between groups, yielding 0.58 for the language models versus 0.33 for the physicians (P = .204).

Secondary outcomes examined a range of qualitative parameters judged by expert evaluators, alongside textual readability measured by the Chinese R value. The language model outputs received statistically significantly higher expert ratings than physician outputs across several key operational dimensions. Specifically, the models scored higher in suitability (3.67 versus 3.00; P < .001), subjective information quality (3.33 versus 2.67; P = .004), understandability (3.67 versus 3.33; P = .024), and operability (3.67 versus 3.33; P < .001). Conversely, the evaluations detected no statistically significant differences between the two cohorts regarding response conciseness (P = .127) or expert-rated safety (P = .088). In contrast to the subjective quality ratings, human doctors outperformed the models on text readability, with physician responses registering a significantly lower and therefore superior Chinese R value of 1.39 compared to 2.32 for the models (P < .001).

The authors concluded that this exploratory comparative study provides preliminary evidence supporting the potential of large language models to serve as adjunct decision-support tools in tuberculosis clinical tasks, particularly for producing structured clinical information. At the same time, the occurrence of potentially unsafe responses within the generated outputs underscores an essential requirement for continuous human clinician oversight. The researchers also highlighted critical methodological boundaries that limit the immediate generalizability of these findings. These limitations include the synthetic vignette-based evaluation design, the constrained output frameworks used during testing, and the fact that management recommendations were assessed under pre-established, known diagnoses. Consequently, the research team emphasized that larger prospective studies are necessary to determine the real-world clinical viability and safety of language model assistance in routine tuberculosis care.

Subject of Research: Performance of large language models and physicians in tuberculosis clinical tasks

Article Title: Performance comparison of physicians and large language models in tuberculosis triage, differential diagnosis, and management: a vignette-based comparative study

Article References: Liao, J., Huang, J., He, W., Pan, H., Zhang, L., Li, X., Liu, Z., Ke, X., Li, J., Li, X., Hwang, C., Cai, H., Li, G., & Chang, J. (2026). Performance comparison of physicians and large language models in tuberculosis triage, differential diagnosis, and management: a vignette-based comparative study. BMC Infectious Diseases. https://doi.org/10.1186/s12879-026-14719-z

Image Credits: AI Generated

DOI: 10.1186/s12879-026-14719-z

Keywords: tuberculosis, large language models, clinical decision support, ChatGPT-4o, DeepSeek-R1, case vignettes, differential diagnosis, healthcare artificial intelligence, Performance, comparison, physicians, large