One Teacher, Many Datasets: New AI Framework Boosts Medical Image Analysis Across Tasks

Researchers have unveiled a unified artificial intelligence framework that pools knowledge from multiple medical imaging datasets and transfers it to compact models capable of segmentation, classification, and object detection.

The team, led by Ciprian-Mihai Ceausescu with Ion-Marian Anghelina and Dumitru-Bogdan Alexe, designed a three-stage pipeline that deliberately separates the aggregation of multi-dataset knowledge from the optimization of any specific task. In the first stage, individual teacher networks are trained with domain-adversarial alignment, a technique in which a domain discriminator pushes the encoder to produce features that make source and target samples indistinguishable. Each source teacher is trained on its own dataset and on the target dataset, encouraging domain-invariant representations. The authors note that in their experiments labeled target annotations were available and exploited, although an unsupervised variant without target labels was also analyzed in an ablation study.

The second stage constructs what the authors call a joint teacher. Feature maps from the target teacher and all source teachers are collected at each resolution level of the encoder and fused through cross-attention blocks, then concatenated along the channel dimension and passed through convolutional residual blocks to harmonize scale and statistics. The encoders and bottleneck layers of all teachers remain frozen during fusion, while a new task-specific head is trained from scratch on the target dataset. This design preserves the domain-specific knowledge captured by each teacher while allowing the joint teacher to specialize to the target task.

In the third stage, knowledge is distilled from the joint teacher into a compact student model using a curriculum-driven objective. The student begins training with a loss weighted primarily toward the supervised task, and a curriculum factor gradually increases the weight of distillation terms, including a contrastive loss that aligns teacher and student similarity distributions, a feature-alignment loss based on mean squared error, and a cosine-similarity loss that encourages directional consistency of feature vectors. This schedule, the authors explain, lets the student first learn a reasonable solution from ground-truth supervision and then progressively refine its features to match the richer, multi-source representation space of the joint teacher.

The framework was evaluated on six segmentation benchmarks spanning MRI and CT: BrainMetShare, ISLES 2022, and BraTS for brain pathologies, and Lung MSD, LiTS, and KiTS for lung, liver, and kidney tumors. Each dataset was treated once as the target while the remaining two datasets of the same modality served as sources, with 80 percent of data used for training, 10 percent for validation, and 10 percent for testing. Performance was measured with the Dice Similarity Coefficient, Intersection over Union, precision, recall, and the 95th percentile Hausdorff distance, which captures boundary accuracy.

The results showed consistent gains for the distilled students over both dataset-specific baselines and multi-head multi-dataset models. With the TResUNet architecture, the student improved average IoU by roughly 1.3 percent on MRI-based tasks and about 2.5 percent on CT-based tasks, with particularly sharp reductions in Hausdorff distance on CT data. With UNet, the average IoU gain on CT datasets reached approximately 4.8 percent. The effect was strongest on BrainMetShare, where numerous small, sparsely distributed metastases benefit from multi-dataset fusion, and on Lung MSD and LiTS, where students achieved IoU gains of about 2.0 and 3.6 points respectively alongside reductions in boundary error of several pixels.

Not every dataset benefited equally. On BraTS, which contains highly heterogeneous glioma scans, the joint teacher underperformed the dataset-specific baseline, and although the student recovered part of the loss, it remained slightly below the dataset-specific model by roughly 0.7 to 0.9 IoU or DSC points. The authors attribute this to the infiltrative and multi-scale nature of gliomas, which limits the direct transfer of representations learned from metastasis and stroke datasets, and they flag this dataset-specific variation as an open problem in identifying when distillation is beneficial. Ablation studies also showed that performance scales with source diversity: a single source dataset yielded modest improvements, while adding a second complementary source nearly doubled the benefit, and combining encoder and bottleneck features for distillation outperformed either level alone.

The framework extended beyond segmentation with minimal modification. On pulmonary disease classification from chest X-rays, using a MedViT-based student, the joint teacher achieved the highest accuracy across COVIDx-CXR, RT-PCR Covid19, and COVID-QU-Ex, with distilled students outperforming both baseline and multi-head approaches; on COVIDx-CXR, the joint teacher reached 83.58 percent accuracy versus 78.54 percent for the dataset-specific baseline, with the best student reaching 81.96 percent. On cerebral degenerative disease classification, the L3 distillation strategy achieved the highest accuracy of 77.76 percent on OASIS MRI, while on ADNI the L2 configuration performed best at 91.25 percent, with performance slightly degrading at deeper distillation, underscoring that the optimal distillation depth is dataset-dependent. For object detection on lung CT datasets using Faster R-CNN and RF-DETR, the joint teacher consistently delivered the strongest performance, and distilled students outperformed dataset-specific and multi-head baselines, for example improving mAP at IoU 0.50 to 0.95 by 3.8 points over the dataset-specific baseline on DeepLesion with Faster R-CNN, while retaining a large fraction of the joint teacher’s gains in a more compact, deployment-friendly model.

The authors are candid about the costs and limitations of the approach. Constructing the joint teacher is the only computationally expensive step, requiring roughly three times the per-iteration training time of a single-dataset baseline, although the distilled student runs at baseline inference speed of approximately 0.0006 seconds per image. The framework also assumes access to multiple labeled source datasets sufficiently related to the target domain, which privacy regulations or data-sharing restrictions may not always permit. All experiments were conducted on a single workstation with an NVIDIA GeForce RTX 3090 GPU, and source and target datasets were grouped by imaging modality, leaving true cross-modality transfer unexplored. The pipeline was validated on a single GPU setup, and the authors note that the framework does not explicitly exploit synergies across tasks with different annotation granularities.

By demonstrating that a single, task-agnostic distillation pipeline can improve generalization across segmentation, classification, and detection, the study offers a template for building medical AI systems that draw on the complementary strengths of heterogeneous clinical datasets. The authors suggest that future work could extend the framework to survival analysis and multi-label diagnosis, incorporate self-supervised pretraining within the joint teacher, and scale the approach to federated and privacy-preserving settings, along with more adaptive fusion mechanisms and uncertainty-aware distillation strategies for challenging clinical environments.

Subject of Research: A multi-dataset knowledge distillation framework for medical image segmentation, classification, and object detection

Article Title: Multi-dataset cross-domain knowledge distillation for unified medical image segmentation, classification, and detection

Article References: Ceausescu, C.-M., Anghelina, I.-M., & Alexe, D.-B. (2026). Multi-dataset cross-domain knowledge distillation for unified medical image segmentation, classification, and detection. Machine Learning with Applications, 26, Article 101052. https://doi.org/10.1016/j.mlwa.2026.101052

Image Credits: AI Generated

DOI: 10.1016/j.mlwa.2026.101052

Keywords: knowledge distillation, medical image segmentation, medical image classification, object detection, teacher-student learning, domain adaptation, cross-attention fusion, MRI, CT, chest X-ray, brain tumor segmentation, lung lesion detection