Multimodal AI for breast cancer diagnosis reviewed from fusion to foundation models

Breast cancer remains one of the most prevalent cancers worldwide and a major cause of cancer-related mortality, and a newly published narrative review argues that the most promising path toward better computer-assisted diagnosis lies not in any single data source, but in combining them. The review, published in Discover Artificial Intelligence, synthesizes multimodal deep learning approaches for breast cancer diagnosis identified through a literature search covering January 2021 to September 2026, and traces the field’s evolution from rule-based statistical models through machine learning and radiomics to today’s vision-language models and foundation models.

The authors, Naoual El Aboudi and Laila Benhlima, searched four primary databases—PubMed, IEEE Xplore, SpringerLink, and ScienceDirect—retrieving 2,224 records before duplicate removal. After applying predefined eligibility criteria, they retained 65 unique publications for detailed analysis. Because the work is a narrative rather than a systematic review, no formal risk-of-bias instrument such as QUADAS-2 was applied, and the authors explicitly state that a complete quantitative breakdown of screened and excluded records cannot be provided. They also caution that reported improvements in model performance should not be interpreted as demonstrated clinical benefit without prospective validation.

The review organizes the field around the data modalities that multimodal systems integrate: mammography, ultrasound, magnetic resonance imaging, histopathology, radiology reports, structured clinical information, and genomic data. Mammography remains the most widely used screening modality, while ultrasound offers complementary information particularly for women with dense breast tissue, and MRI provides superior soft-tissue contrast. Histopathological imaging serves as a reference standard for confirming diagnosis. Genomic resources such as METABRIC and TCGA-BRCA have enabled imaging-genomic studies targeting molecular subtype classification, risk assessment, and survival prediction, although the authors note that patient-level pairing of imaging and genomic data remains limited and must be reported explicitly.

A central analytical thread is the taxonomy of fusion strategies. Early fusion combines features from different modalities before learning begins, which is simple and computationally efficient but limited in capturing complex nonlinear relationships. Intermediate, or feature-level, fusion analyzes each modality through dedicated encoders before combining representations, often using attention mechanisms, multimodal transformers, or joint embedding spaces. Late fusion integrates predictions from independently trained modality-specific models through voting, weighted averaging, or ensembling, offering robustness to missing data but potentially underexploiting cross-modal dependencies. Hierarchical fusion combines information at multiple stages of the learning process and is considered among the most effective strategies for advanced systems, though it is computationally demanding and requires larger annotated datasets.

The review identifies imaging-report fusion as one of the most recent and promising research directions. Radiology reports contain rich textual descriptions of imaging findings, BI-RADS assessments, and lesion characteristics that complement visual features. A highlighted contribution is Mammo-CLIP, a CLIP-style vision-language model specifically designed for mammography, pretrained on large-scale mammogram-report pairs and reported to improve BI-RADS-aware classification, lesion localization, and cross-dataset generalization within the original study’s validation setting. Another framework, ManiNeg, incorporates radiological manifestations extracted from reports to guide multimodal representation learning. On the generative side, MammoVLM is described as a multimodal assistant capable of answering mammography-related questions and generating explanations, emphasizing human-AI interaction over pure classification.

Beyond breast-specific systems, the review surveys general biomedical foundation models whose relevance to breast cancer lies in potential transfer. BiomedCLIP, pretrained on approximately 15 million biomedical image-text pairs from scientific literature, offers a potentially useful representation but was not developed specifically for mammography and requires task-specific evaluation. MultiModN is a modular framework able to operate when some modalities are unavailable, addressing the missing-modality problem common in real-world clinical data. CPath-Omni, a large-scale computational pathology model, is potentially relevant because histopathology is central to tumor characterization, but the authors stress that its broader capabilities should not be conflated with breast-specific diagnostic performance unless such evaluation has been reported.

The review is candid about methodological limitations across the literature. Most studies are trained and evaluated on relatively small or single-institution retrospective datasets, and external validation with independent multicenter cohorts remains limited. Many multimodal models assume all modalities are simultaneously available, yet missing modalities frequently occur in routine practice. Fairness across patient populations, breast density categories, and imaging devices is rarely assessed, and comparatively little attention has been paid to calibration and predictive uncertainty estimation, which clinicians need alongside the diagnosis itself. Direct comparison of reported accuracy, AUC, sensitivity, and F1-score across studies is also problematic because datasets, preprocessing pipelines, and validation protocols differ substantially.

Data availability emerges as a recurring barrier. While public mammography datasets such as CBIS-DDSM, INBreast, VinDr-Mammo, and CMMD are widely used, resources that simultaneously include imaging, clinical information, genomic profiles, and radiology reports remain scarce. Paired image-report datasets are especially rare because of privacy regulations and institutional restrictions, pushing many vision-language studies toward proprietary data and limiting reproducibility and fair benchmarking. The review also corrects a common confusion: MV-MLM, a multi-view mammography and language framework, is a modeling approach rather than a public dataset, an important distinction when assessing the true availability of paired mammography-report resources.

Looking forward, the authors outline several research priorities. These include breast-specific multimodal foundation models integrating mammography, ultrasound, MRI, pathology images, reports, clinical information, and genomic data within unified frameworks; retrieval-augmented systems that consult similar cases, guidelines, and literature during inference to reduce hallucinations; explainable multimodal learning combining saliency maps, concept attribution, and natural-language explanations; federated learning enabling multicenter collaboration without sharing raw patient data; and synthetic data generation, which requires careful validation of clinical realism and consistency. Real-world deployment will additionally demand interoperability with PACS, RIS, and electronic health records, regulatory compliance, and clinician acceptance.

The review’s overarching message is one of tempered optimism. Vision-language and foundation models represent a genuine transition from task-specific classifiers toward flexible, transferable multimodal systems that could support lesion detection, BI-RADS assessment, report generation, and clinical decision support. But the authors repeatedly emphasize that current evidence is predominantly retrospective and technical, that these applications remain emerging research directions rather than established clinical capabilities, and that robust prospective multicenter validation, standardized benchmarking, and transparent explainability are required before multimodal AI can be routinely integrated into breast cancer diagnosis—with final decisions remaining the responsibility of qualified healthcare professionals.

Subject of Research: Multimodal deep learning approaches for breast cancer diagnosis

Article Title: Recent advances in multimodal deep learning for breast cancer diagnosis using data fusion vision language models and foundation models

Article References: El Aboudi, N., & Benhlima, L. (2026). Recent advances in multimodal deep learning for breast cancer diagnosis using data fusion vision language models and foundation models. Discover Artificial Intelligence, 6(1), Article 1657. https://doi.org/10.1007/s44163-026-02625-x

Image Credits: AI Generated

DOI: 10.1007/s44163-026-02625-x

Keywords: breast cancer, multimodal deep learning, data fusion, vision-language models, foundation models, mammography, radiology reports, image-text fusion, clinical validation, artificial intelligence, Recent, advances