Large language models can deliver accurate and clinically safe responses to questions regarding chronic pancreatitis, but their breadth of coverage diverges from that of experienced medical professionals, according to a comparative investigation published in BMC Gastroenterology. The study assessed the performance of DeepSeek-V3 and ChatGPT-4o alongside physician responses from established online medical platforms, analyzing both guideline-based knowledge and real-world patient concerns. While the artificial intelligence platforms matched conventional sources on fundamental safety and core accuracy, variations emerged in the breadth of information provided.
Chronic pancreatitis is a complex, progressive inflammatory disorder marked by irreversible destruction of the pancreatic parenchyma, which frequently culminates in severe exocrine and endocrine dysfunction. Individuals diagnosed with the condition often endure debilitating chronic pain, malabsorption, and diabetes mellitus, complications that profoundly undermine quality of life. Effective clinical management relies heavily on accessible, reliable, and up-to-date medical guidance for patients. In recent years, generative artificial intelligence systems have increasingly been consulted by the public as accessible medical knowledge tools, prompting researchers to rigorously evaluate their reliability against traditional physician-led resources.
To establish a rigorous benchmark, the investigators designed an evaluation centered on two distinct domains: clinical guidelines and genuine patient experiences. The first domain drew upon evidence-based recommendations established in the 2020 American College of Gastroenterology Clinical Guidelines. The second domain captured direct, patient-specific concerns compiled from real-world clinical follow-up records. By examining both formalized medical consensus and practical patient inquiries, the research aimed to delineate the specific capabilities and operational boundaries of large language models when used for patient education and clinical decision support.
The evaluation was executed across two sequential phases using 21 guideline-derived questions and 40 patient-derived inquiries. During the initial phase, two blinded gastroenterologists independently graded the outputs generated by DeepSeek-V3 and ChatGPT-4o across four primary dimensions: accuracy, completeness, comprehensiveness, and safety. The specialist scores were subsequently averaged for each specific query and model. In the second phase, answers from both computational models were benchmarked directly against physician answers retrieved from online medical platforms, with 40 patients rating all three information sources for communication empathy and overall satisfaction.
Statistical assessments utilized paired Wilcoxon signed-rank tests or Friedman tests followed by paired Wilcoxon tests adjusted via the Holm procedure. Because individual participant-level data were unavailable for secondary reanalysis, patient-reported metrics were examined through matched question-level aggregated means. When addressing the 21 guideline-based inquiries, DeepSeek-V3 and ChatGPT-4o exhibited comparable performance across accuracy, completeness, comprehensiveness, and safety, showing that both models were equally adept at parsing formal medical standards.
In the evaluation of the 40 real-world patient inquiries, all three information sources maintained comparable levels of accuracy and safety. Although an initial omnibus analysis revealed an overall difference in completeness (P = 0.015), post-hoc pairwise comparisons showed no statistically significant differences following Holm adjustment. However, comprehensiveness differed significantly across the sources (P < 0.001). DeepSeek-V3 attained lower comprehensiveness ratings compared to online medical platforms (Holm-adjusted P = 0.020) and ChatGPT-4o (Holm-adjusted P = 0.009). By contrast, no statistically significant difference in comprehensiveness was observed between ChatGPT-4o and the online medical platforms (Holm-adjusted P = 0.499).
Assessments of patient-centered communication demonstrated no statistically significant differences among the three sources regarding perceived empathy or overall user satisfaction. Across all evaluated cohorts, ratings for empathy were comparatively favorable, whereas overall satisfaction ratings remained modest. The researchers concluded that while DeepSeek-V3 and ChatGPT-4o produce broadly accurate and safe medical guidance, a model’s high descriptive completeness does not inherently ensure expansive clinical context. Consequently, artificial intelligence tools remain best suited as educational supplements rather than replacements for individualized physician assessments.
Subject of Research: Comparative evaluation of large language models and online medical platforms addressing chronic pancreatitis inquiries
Article Title: Comparison of Deepseek-V3, ChatGPT-4o, and online medical platforms in addressing chronic pancreatitis related questions
Article References: Li, W., Yang, Z., Qiu, G., Su, X., Yang, Z., Zhu, H., Liu, C., Zhang, D., Huang, H., & Jin, Z. (2026). Comparison of Deepseek-V3, ChatGPT-4o, and online medical platforms in addressing chronic pancreatitis related questions. BMC Gastroenterology. https://doi.org/10.1186/s12876-026-05446-5
Image Credits: AI Generated
DOI: 10.1186/s12876-026-05446-5
Keywords: Chronic Pancreatitis, DeepSeek-V3, ChatGPT-4o, Large Language Models, Online Medical Platforms, Artificial Intelligence in Medicine, Patient Education, Clinical Decision Support, American College of Gastroenterology, Gastroenterology, Medical Information Accuracy, Clinical Safety, Comprehensiveness, Patient Satisfaction, Perceived Empathy, BMC Gastroenterology
News Source: Ophelia Keating. (November 12, 2026). AI Models Match Physicians on Chronic Pancreatitis Safety but Differ in Comprehensiveness. Scienmag.



