Research report: Comparing teacher-made and AI-assisted teaching videos for thyroid eye disease: an exploratory study

In a small randomized study at Shanghai Changzheng Hospital, resident doctors who watched a five-minute, expert-curated, AI-assisted video about thyroid eye disease showed significant knowledge gains five minutes later, but their improvement from baseline was not statistically significant one week later. Residents who watched a teacher-made video had a similar score trajectory. The study detected no statistically significant differences between the groups at baseline, five minutes or one week, although its design and sample size did not establish that the two formats were equivalent.

Producing the AI-assisted video required more human effort than making the teacher version and did not reduce total production time. The exploratory study enrolled general and ophthalmology residents at one hospital, where limited access to thyroid eye disease specialists has been identified as an obstacle to training. Thyroid eye disease, also called thyroid-associated ophthalmopathy, is a progressive autoimmune disorder involving orbital inflammation.

Its manifestations can include eyelid retraction, swelling, protrusion of the eyes, restricted eye movement and double vision, while dysthyroid optic neuropathy is an acute, vision-threatening complication. The investigators included both general and ophthalmology residents because general practitioners may be involved in recognizing the condition early, and because DON requires timely diagnosis and intervention. Researchers initially recruited 32 general residents and 20 ophthalmology residents in November 2025.

Eligibility required written consent and less than five hours of prior systematic thyroid eye disease training. Participants completed a baseline online test containing 20 single-choice questions on anatomy, clinical manifestations, diagnosis and treatment. They were then stratified as general or ophthalmology residents and randomly assigned in a 1:1 ratio to watch either an AI-assisted or teacher-made video.

The final primary analysis included 46 participants: 23 in each group, comprising 28 general residents and 18 ophthalmology residents. Two participants were excluded for failing to complete all three rounds and four for completing a test in less than 120 seconds. The video-production methods differed substantially.

For the AI-assisted version, baseline questionnaire data were shared with the DeepSeek-V3 language model and two orbital-disease specialists to identify knowledge gaps. DeepSeek generated a script, subtitles and text intended to guide illustrations. Those materials were supplied to the Dreamina platform, which generated images and video segments, and the selected outputs and AI-assisted voiceovers were assembled in CapCut.

The generated visuals often showed substantial variability and occasional inappropriate or distorted content. Specialists spent an additional 2.5 hours correcting content-specific errors, replacing distorted images, adjusting voiceover pacing and adding annotations, bringing estimated total human effort for the AI-assisted video to approximately 6.5 hours. The teacher-made video took 4.5 hours to produce.

Both final videos addressed thyroid eye disease anatomy, pathogenesis, clinical manifestations, differential diagnosis and treatment. The AI-assisted version ran for 5 minutes 8 seconds, while the teacher-made version lasted 5 minutes 12 seconds. Six experts—four senior ophthalmologists specializing in thyroid eye disease and two medical-education specialists—rated each video across clinical accuracy, visual fidelity and pedagogical appropriateness.

Eleven of 12 AI-video items met the prespecified item-level content-validity threshold of 0.78. The exception concerned whether pathological changes were represented accurately and without distortion, for which the item-level index was 0.67. All 12 teacher-video items met the threshold, with values from 0.83 to 1.00.

The scale-level averages were 0.93 for the AI-assisted video and 0.97 for the teacher-made video, and mean ratings were 4.43 and 4.56 out of 5, respectively. Learning was evaluated through three 20-item assessments delivered through a Chinese online survey platform. Test 1 measured baseline knowledge, Test 2 was administered five minutes after the video to assess immediate learning, and Test 3 was administered one week later to assess delayed retention.

The researchers created parallel forms with a planned mix of 35% easy, 35% moderate and 30% difficult questions, based on correct-response rates from the Chinese Ophthalmology Residency Training Examination. Six thyroid eye disease experts also assessed the questions’ content validity. The Kuder–Richardson Formula 20 reliability coefficients were 0.80 for the baseline test, 0.76 for the five-minute test and 0.82 for the one-week test, indicating acceptable to good internal consistency.

Participants and instructors could not be blinded to video type, but the data analysts were blinded to group assignments. In the AI-assisted group, mean test scores increased from 65.65 at baseline to 78.70 five minutes after viewing. The estimated mean difference was 13.04 points, with a 95% confidence interval of 3.51 to 22.57, a Bonferroni-adjusted p value of 0.004 and a within-group effect size of Cohen’s d_rm = 0.61.

At one week, the mean score was 73.26. The 7.61-point difference from baseline was not statistically significant, with a 95% confidence interval of −1.92 to 22.57 and p = 0.16. In the teacher-made group, the mean score rose from 68.04 at baseline to 79.57 five minutes later, a significant increase of 11.52 points with a confidence interval of 1.99 to 21.05 and p = 0.01.

Its four-point increase from baseline at one week was also not statistically significant, with a confidence interval of −4.75 to 14.31 and p = 0.67. Between-group score differences were nonsignificant at all three assessments: the AI-assisted-minus-teacher-made mean difference was −4.64 points at baseline, −3.12 points at five minutes and −1.81 points at one week. The group-by-time interaction was also nonsignificant, indicating no detected difference in score trajectories between the groups.

Completion times showed a different within-group pattern for the AI-assisted participants. Median time fell from 522 seconds at baseline to 347 seconds five minutes later and 253 seconds one week later. In the adjusted mixed-effects analysis, the follow-up-to-baseline geometric mean ratios were 0.57 at five minutes, with a 95% confidence interval of 0.39 to 0.82, and 0.45 at one week, with a confidence interval of 0.32 to 0.66.

No significant within-group time changes were detected in the teacher-made group, and between-group differences were nonsignificant at both follow-ups. Because the study included no no-video control group, changes over time cannot be attributed causally to either educational video. The researchers also conducted an intention-to-treat analysis including all 52 randomized participants and a sensitivity analysis of 50 participants who completed the tests; neither analysis materially changed the pattern of nonsignificant between-group results.

Performance on questions about dysthyroid optic neuropathy did not improve significantly across the three rounds with either format. Exploratory comparisons between ophthalmology and general residents produced numerical differences, but the subgroups were small and the differences were not formally tested. A clearer association emerged between participants’ perceived learning experience and learning outcome, which had a Spearman correlation of 0.60 and p below 0.001, whereas optimism about AI was not significantly correlated with learning outcomes.

Participants generally rated the instructional materials favorably. The AI-assisted video was described as high quality, while the teacher-made video was considered generally acceptable. Most items concerning learning outcomes, the learning experience and optimism about AI received average ratings above 4 on a five-point scale.

Residents nevertheless remained cautious about factual reliability: agreement with the statement that AI-tool content is completely accurate averaged 3.76. In optional open responses, 13 of 27 respondents, or 48.1%, identified pacing as the aspect most needing improvement; 9 mentioned content comprehensiveness and 7 visual quality, with respondents able to report more than one theme. Complaints about the AI-assisted video included stiff animation and insufficient visual highlighting of key terms, whereas the teacher-made video drew criticism for rigid visuals and narration, weak labeling, abrupt animation, rapid speech and occasional pronunciation errors.

Several limitations constrain the findings. The sample was small and came from a single hospital, and the sample-size calculation was designed around within-group improvement in the AI-assisted group rather than a statistically powered comparison between groups. No non-inferiority margin was prespecified, meaning the absence of a statistically significant difference does not show that the formats are equivalent.

The three test forms were not formally equated, although their item difficulty and discrimination measures were broadly comparable, and follow-up ended after one week. Learning outcomes were assessed with knowledge tests rather than clinical performance, direct observation of reasoning or objective structured clinical examinations. Online participation also left data quality and engagement incompletely controlled, while AI-specific factual errors were reviewed by experts but not systematically quantified.

Subject of Research: Exploratory randomized comparison of AI-assisted and teacher-made thyroid eye disease teaching videos for medical residents

Article Title: Comparing teacher-made and AI-assisted teaching videos for thyroid eye disease: an exploratory study

Article References: Liu, S., Wang, Y., Chen, Y., Chang, Y., Wang, Y., Gao, Y., & Wei, R. (2026). Comparing teacher-made and AI-assisted teaching videos for thyroid eye disease: an exploratory study. Frontiers in Cell and Developmental Biology, 14, Article 1963967. https://doi.org/10.3389/fcell.2026.1963967

Image Credits: AI Generated

DOI: 10.3389/fcell.2026.1963967

Keywords: thyroid eye disease, generative AI, medical education, ophthalmology residents, teaching videos, randomized study, knowledge retention, expert review, dysthyroid optic neuropathy, Comparing, teacher-made, AI-assisted