AI Scores Health Videos on Douyin Nearly as Well as Human Experts, Study Finds

Short-form video platforms have quietly become one of the most influential sources of health information in the world, and China’s Douyin is no exception. For new mothers searching for guidance on lactational mastitis, a painful and potentially serious inflammation of breast tissue that affects a substantial proportion of breastfeeding women, the platform offers thousands of clips promising advice on symptoms, treatment, and prevention. The problem, as clinicians have long suspected, is that the quality of that advice varies enormously. A new study published in BMC Nursing by researchers at the Women’s and Children’s Hospital of Chongqing Medical University and their collaborators now offers both a sobering snapshot of that content landscape and a strikingly efficient technological solution for policing it.

The research team, led by Siyu Zhou and Junfeng Li as co-first authors with Ben Li as corresponding author, set out to answer two intertwined questions. First, how reliable and high-quality are the lactational mastitis videos that new parents actually encounter on Douyin? Second, could a large language model such as ChatGPT-4, carefully adapted for the task, evaluate that content as reliably as trained human experts, and do so at a scale and speed that manual review could never match? The answer to the second question, according to their data, appears to be a qualified but impressive yes.

To build their dataset, the researchers searched Douyin, the Chinese short-video platform operated separately from TikTok, for content related to lactational mastitis posted between January 2021 and June 2025. After strict screening, 50 videos made the final sample. That number may sound modest, but it reflects the deliberate rigor of the selection process: the team needed videos that genuinely addressed the condition as a health education topic, and each one had to be scored in detail by both human raters and the artificial intelligence system. Fifty deeply assessed videos, the authors argue, provide a meaningful proof of concept for a framework that could later be scaled to thousands.

The methodological heart of the study is a hybrid instrument the authors call AI-DISCERN. The DISCERN instrument is a well-established, validated tool originally designed to assess the quality of written health information, probing whether a publication has clear aims, reliable sources, balanced treatment of uncertainty, and realistic descriptions of benefits and risks. Adapting it for video content and for the specific concerns of nursing education, the researchers integrated ChatGPT-4 into the scoring pipeline, refined the framework through nursing-oriented adaptation, and pre-tested it until the agreement between AI-assisted scores reached an intraclass correlation coefficient of 0.923, a level conventionally interpreted as excellent.

As a reference standard, two trained nursing experts independently scored every video while blinded to each other’s judgments and to the AI’s outputs. The human raters themselves agreed remarkably well, with an intraclass correlation coefficient of 0.976 for the total score, confirming that the assessment task was well defined and consistently executable. When the AI’s scores were compared against the human consensus, the agreement was even stronger: an intraclass correlation coefficient of 0.994, with a 95 percent confidence interval of 0.989 to 0.996. On twelve individual items of the instrument, the weighted Kappa coefficient exceeded 0.80, again signaling near-perfect categorical agreement between machine and expert.

Speed is where the technology truly separates itself from human effort. The AI system required an average of 33.34 seconds, with a standard deviation of 7.29 seconds, to score a single video. Human experts needed 89.79 seconds on average, with a standard deviation of 42.09 seconds, making the AI approximately 2.69 times faster, a difference that was statistically significant at P less than 0.001. That gap may seem trivial for 50 videos, but it compounds dramatically at platform scale. A repository of 100,000 health videos that would demand months of expert labor could, in principle, be triaged by an AI-assisted framework in a fraction of the time, freeing clinicians to focus on the borderline cases and on crafting better educational content themselves.

The study’s descriptive findings about who creates mastitis content on Douyin are equally noteworthy. A striking 86 percent of the included videos were posted by professional medical staff, and another 8 percent by professional medical institutions, while only three videos, or 6 percent, came from non-professional sources. This suggests that on this particular health topic, credentialed voices dominate the supply side of the platform. Yet the researchers observed, descriptively, that non-professional videos appeared to attract higher engagement indicators. Because the non-professional subgroup contained only three videos, the authors explicitly flagged any findings involving that comparison as exploratory, a caution that reflects sound statistical practice rather than a definitive claim about what goes viral.

Perhaps the most counterintuitive result concerns the relationship between quality and popularity. The total DISCERN score was positively correlated with video duration, with a Spearman rank correlation coefficient of 0.498 and a P value below 0.001, meaning that longer videos tended to be more thorough and reliable. However, the total quality score showed no significant correlation with any of the engagement indicators, such as likes or shares, with all P values exceeding 0.05. In other words, the algorithmic and social forces that determine which videos get watched do not appear to reward educational quality in this sample. A parent scrolling Douyin cannot assume that a highly engaged video is a trustworthy one, and a well-made, evidence-based video has no inherent visibility advantage over a superficial competitor.

For clinicians and public health communicators, that disconnect between quality and engagement is the study’s most actionable insight. If popularity does not track reliability, then platforms, professional bodies, and educators cannot rely on organic market forces to surface the best content. Structured, scalable quality assessment becomes essential infrastructure. The AI-DISCERN framework, the authors suggest, may serve as a feasible supportive tool for exactly that purpose: a way to systematically grade lactational mastitis education, flag substandard material, and identify gaps that professional creators should fill. It is a supportive tool, not a replacement for human judgment, and the authors are careful on this point.

The limitations are clearly acknowledged. The sample of 50 videos is small, drawn from a single platform, a single language, and a single clinical topic, and the exploratory findings about non-professional content rest on just three videos. The authors state that further external validation using larger and more diverse datasets is needed before broader application. The study analyzed only publicly available videos without collecting personally identifiable information, and the ethics committee of Chongqing Health Center for Women and Children determined that formal ethical approval and informed consent were exempt under Guideline 22 of the International Ethical Guidelines for Health-related Research Involving Humans. Funded by a 2025 hospital-level medical education research grant from the Women and Children’s Hospital of Chongqing Medical University, the work points toward a future in which large language models act as tireless, calibrated reviewers of the health content that billions of people consume, one short video at a time.

Subject of Research: AI-assisted evaluation of the quality and reliability of lactational mastitis health education videos on Douyin

Article Title: ChatGPT-assisted evaluation of lactational mastitis videos on Douyin: quality and reliability as a health education resource

Article References: Zhou, S., Li, J., Tang, F., Tang, X., Shi, Y., An, L., & Li, B. (2026). ChatGPT-assisted evaluation of lactational mastitis videos on Douyin: quality and reliability as a health education resource. BMC Nursing. https://doi.org/10.1186/s12912-026-05282-8

Image Credits: AI Generated

DOI: 10.1186/s12912-026-05282-8

Keywords: ChatGPT, lactational mastitis, Douyin, short videos, health education, DISCERN instrument, patient education, artificial intelligence, nursing, social media, quality assessment, misinformation