AI learns to map children’s swallowing videos from sparse expert labels

Researchers have developed a deep learning framework that segments key anatomical structures and swallowing events in pediatric fiberoptic endoscopic evaluation of swallowing (FEES) videos, even when trained on sparse annotations from just five expert clinicians. The study, published in Medical & Biological Engineering & Computing, compared supervised U-Net-based models against a training-free, in-context video segmentation approach and found that both could identify structures in previously unseen pediatric patients, though performance dropped notably outside the training data.

Accurate analysis is essential for detecting swallowing abnormalities in children, but automated interpretation has been hampered by three persistent obstacles: high variability in how different experts annotate the same video frames, severe class imbalance between common and rare findings, and the limited availability of labeled pediatric datasets. The new work addresses these challenges directly by evaluating what can be achieved with minimal annotation effort.

The team evaluated three approaches. The first was a class-wise U-Net++ model, a supervised convolutional architecture widely used for medical image segmentation. The second was VM-UNet, a more recently proposed architecture built around a vision Mamba backbone. The third, called Temporal, was a pretrained video object segmentation framework applied in a training-free, in-context learning setting, meaning it relied on retrieval from annotated examples rather than on weight updates from the target dataset. Six classes were considered: laryngeal inlet, vocal cords, penetration, residue, spillage, and salivary pooling.

Two complementary experiments were performed. The first was an internal frame-level five-fold cross-validation on 38 videos. The second was a stricter independent patient-level evaluation, in which models trained on the 38 videos were tested on 50 newly annotated videos from entirely different patients. This design matters because frame-level validation on the same patients can substantially inflate apparent performance, particularly for retrieval-based in-context models that can match similar frames from the same recording.

In the internal cross-validation, mean Dice scores were 0.70 for U-Net++, 0.69 for VM-UNet, and 0.77 for Temporal. On the independent test set, however, mean video-level Dice across five classes fell to 0.45 for U-Net++, 0.44 for VM-UNet, and 0.40 for Temporal. Laryngeal inlet segmentation reached up to 0.67 Dice, and a presence score for that structure reached 0.96, indicating that detecting whether the structure appears in a frame is considerably easier than delineating its exact boundaries.

The results reveal a clear pattern: supervised models were more accurate for anatomical structures, whereas all approaches performed comparably for swallowing events such as penetration, residue, and spillage. Penetration, the passage of material into the laryngeal inlet without entering the airway below the vocal cords, remained the most challenging class, consistent with it having the lowest inter-expert agreement among the five annotators.

A key finding concerns validation methodology. Frame-level cross-validation substantially overestimated performance, particularly for the retrieval-based in-context model, which benefits from temporally adjacent frames sharing the same patient and recording conditions. The patient-level independent test provides a more realistic estimate of how such systems would behave when deployed on new patients, and the gap between the two evaluations underscores the importance of patient-level splits in medical video segmentation research.

Despite the performance drop on unseen patients, the authors conclude that models trained on sparse, mostly single-expert annotations can segment anatomical structures in unseen pediatric patients. The comparison also suggests that training-free in-context segmentation, which requires no model retraining, can approach supervised performance for swallowing events, offering a potentially practical alternative when annotation budgets are tight.

The work contributes a pediatric FEES segmentation framework and a head-to-head comparison of supervised and training-free paradigms under realistic annotation constraints. As automated FEES analysis moves toward clinical use, the findings highlight both the promise of learning from limited expert input and the caution needed when interpreting frame-level validation numbers, especially for in-context models that can exploit temporal redundancy within a single patient’s recording.

Subject of Research: Deep learning segmentation of pediatric swallowing endoscopy videos using sparse multi-annotator data

Article Title: Deep learning-based pediatric swallowing video segmentation from sparse multi-annotator endoscopic data: comparing supervised U-Net models and in-context segmentation

Article References: Abbasi, S., Al-Kassem, H., Firouzi, Z., AlKadi, M., Kurian, M., Adsett, A., Rappazzo, C., El-Hakim, H., Jaremko, J., & Hareendranathan, A. (2026). Deep learning-based pediatric swallowing video segmentation from sparse multi-annotator endoscopic data: comparing supervised U-Net models and in-context segmentation. Medical & Biological Engineering & Computing. https://doi.org/10.1007/s11517-026-03703-z

Image Credits: AI Generated

DOI: 10.1007/s11517-026-03703-z

Keywords: pediatric dysphagia, FEES, video segmentation, deep learning, U-Net, in-context learning, medical imaging, annotation variability, Dice coefficient, swallowing events, Deep, learning-based