Multi-Backbone Model Classifies Brain Tumors Across MRI Datasets Without External Validation

Computer science researchers have introduced a multi-backbone deep learning framework called CAFM v4 that integrates convolutional and transformer encoders to classify brain tumors from magnetic resonance imaging scans, achieving high benchmark accuracies across five independent datasets. In a study published in Frontiers in Artificial Intelligence, investigators Nagamani Kamineni and R. K. Nadesh from the Vellore Institute of Technology evaluated the computational architecture on image collections ranging from binary tumor detection to a thirty-class challenge combining tumor types and imaging sequences. The model reached test accuracies of 99.78 percent on the binary Br35H dataset, 98.01 percent on the three-class FigShare dataset, 96.75 percent on a four-class collection, 95.79 percent on a fifteen-class dataset, and 97.73 percent on the thirty-class benchmark. Despite these high numerical scores, statistical comparisons revealed that the multi-branch architecture did not achieve statistically significant superiority over its single primary convolutional backbone, and the authors cautioned that the benchmark evaluations do not establish external or cross-institutional clinical generalization.

Accurate identification and classification of brain and central nervous system tumors are fundamental components of diagnosis and treatment planning in neuro-oncology, where clinicians must distinguish between diverse tumor pathologies such as gliomas, meningiomas, and pituitary adenomas. Gliomas represent the most frequent primary brain neoplasms, accounting for approximately 78 percent of all malignant brain tumors in adult populations, while meningiomas typically develop slowly from tissues surrounding the brain and spinal cord. In clinical practice, magnetic resonance imaging serves as the preferred diagnostic modality over computed tomography, positron emission tomography, and X-ray examinations because it avoids exposing patients to ionizing radiation and yields superior soft-tissue contrast across varying acquisition sequences such as T1-weighted, contrast-enhanced T1-weighted, T2-weighted, and fluid-attenuated inversion recovery scans. However, manual interpretation of complex MRI scans by specialized neuroradiologists remains labor-intensive and subject to inter-observer variability, motivating the development of automated computer-aided diagnosis systems to assist clinicians in processing imaging data efficiently.

Recent advances in automated medical image analysis have leveraged convolutional neural networks, vision transformers, and hybrid multi-branch architectures to extract diagnostic representations from radiological scans. Standard convolutional networks utilize small learnable filters across successive layers to capture localized visual primitives such as edges, textures, and tissue boundaries, whereas vision transformers split images into discrete patches and compute self-attention to capture long-range contextual relationships across different anatomical regions. The hierarchical Swin Transformer computes self-attention within local windows that shift between consecutive layers, enabling efficient spatial and contextual modeling, while modern convolutional networks like ConvNeXt V2 introduce global response normalization to refine visual representations. Prior studies attempting to combine these complementary paradigms frequently faced challenges, including architectural imbalances among multi-branch representations, an absence of explicit pairwise interactions during feature fusion, and limited evaluations restricted to single or small benchmark datasets.

To address these architectural limitations, the researchers engineered the Cross-Attention Fusion Module version 4, designated as Anchor-Dominant Residual Fusion, which operates across three distinct parallel backbone encoders. The framework establishes ConvNeXt V2 Base, comprising 87.7 million parameters, as the central anchor branch responsible for providing the primary native 1,024-dimensional feature representation. Concurrently, Swin-Base, containing 86.7 million parameters, generates a 1,024-dimensional feature vector capturing hierarchical spatial context, while EfficientFormerV2-S0, a lightweight mobile-focused transformer containing 3.2 million parameters, contributes a computationally efficient 176-dimensional auxiliary representation. In the third stage of the pipeline, branch projectors map all three disparate backbone outputs into a shared 512-dimensional embedding space, ensuring uniform dimensionality before entering the fusion mechanism.

The defining innovation of the ADRF module is an asymmetric multi-head cross-attention mechanism wherein the projected ConvNeXt V2 representation serves as the query, while the concatenated Swin-Base and EfficientFormerV2-S0 features act as the joint key and value representations. This configuration allows the auxiliary transformer branches to generate an adaptive correction feature that is integrated with the ConvNeXt V2 anchor stream through an explicit residual connection. The cross-attention correction is modulated by a sample-specific and feature-specific 512-dimensional learnable gate, which was initialized to 0.10 and iteratively optimized through backpropagation during model training. In parallel with this anchor-dominant residual pathway, a global multi-head self-attention module processes the three branch projections to capture broader inter-branch contextual relationships, after which the concatenated representations pass through layer normalization, dropout, and Gaussian error linear unit activation layers into a multi-layer perceptron classification head.

The complete training pipeline encompassed an optimization framework governed by four additive loss components and a four-phase progressive learning schedule. The overall objective integrated a primary class-balanced focal loss—configured with class-specific focusing parameters ranging from 2.0 to 4.0 to counter severe class imbalances—alongside two auxiliary cross-entropy losses attached to the Swin-Base and EfficientFormerV2-S0 branches to supply deep supervision during backpropagation. Additionally, a self-distillation loss using Kullback–Leibler divergence at a temperature of 4.0 transferred discriminative representations from the ConvNeXt V2 branch to the final classifier, contributing 40 percent of the total training loss. Model parameters were optimized incrementally: Phase 1 froze all backbones while warming up the projection layers and classification heads; Phase 2 unfroze the ConvNeXt V2 anchor for targeted fine-tuning; Phase 3 engaged all backbones for end-to-end full-network optimization; and Phase 4 implemented stochastic weight averaging to enhance training stability and prevent reliance on an isolated checkpoint.

The authors evaluated the CAFM v4 framework across five public benchmark MRI collections sourced from Kaggle, encompassing varying sample sizes, acquisition protocols, and diagnostic complexities. The binary Br35H dataset contained 3,000 images divided evenly between tumor and non-tumor cases, while the FigShare dataset comprised 3,064 T1-weighted contrast-enhanced images spanning glioma, meningioma, and pituitary tumors. The four-class collection contained 7,200 images, and the fifteen-class dataset encompassed 4,455 images covering diverse neoplasms alongside non-neoplastic inflammatory lesion mimics such as tuberculoma and granuloma. The largest evaluation, the thirty-class dataset, consisted of 11,300 images structured across thirty distinct diagnosis-sequence categories covering ten core pathologies imaged under T1, contrast-enhanced T1, and T2 weightings. Preprocessing standardized inputs to 224 by 224 pixels with three channels, incorporating data augmentations such as gamma correction, Rician noise, flips, elastic deformations, Mixup, and CutMix during training, followed by eight-view or twelve-view test-time augmentation during validation and testing.

Experimental outcomes on held-out test partitions demonstrated high classification performance across all five datasets under a validated checkpoint-selection protocol. On the binary Br35H test set of 450 images, CAFM v4 achieved 99.78 percent accuracy, 99.78 percent macro F1-score, and an area under the receiver operating characteristic curve of 1.0000, misclassifying only one non-tumor image. On the FigShare dataset, testing across 402 images yielded 98.01 percent accuracy, a 97.62 percent macro F1-score, and an AUC-ROC of 0.9987. For the four-class dataset, the model correctly classified 1,548 of 1,600 test images, attaining 96.75 percent accuracy, a 96.70 percent macro F1-score, and an AUC-ROC of 0.9924. In the highly imbalanced fifteen-class task, the model achieved 95.79 percent accuracy and a 95.22 percent macro F1-score across 594 test samples, while the thirty-class diagnosis-sequence task produced 97.73 percent accuracy and a 0.9794 macro F1-score across 1,497 test images.

Analysis of the learnable ADRF fusion gate revealed dataset-specific convergence values rather than a universal weighting scheme across different classification tasks. Starting from an initial bias corresponding to 0.1000, the effective gate parameter converged to 0.1031 on Br35H, 0.1036 on FigShare, 0.1071 on the four-class dataset, 0.1219 on the fifteen-class dataset, and 0.1165 on the thirty-class benchmark. These small upward adjustments indicated that while the ConvNeXt V2 anchor retained primary influence over the fused representation, the auxiliary transformer branches contributed subtle, task-dependent corrections that proved especially impactful in challenging tasks. Ablation experiments underscored this dependency: removing the ADRF module caused only modest performance decreases on simpler datasets but triggered a steep accuracy drop on the fifteen-class benchmark from 95.79 percent down to 69.02 percent, confirming the vital regularizing role of multi-branch cross-attention under severe class imbalance.

Despite the high absolute scores, formal paired statistical testing using McNemar tests indicated that CAFM v4 did not achieve a statistically significant performance advantage over a standalone ConvNeXt V2 anchor evaluated under identical test conditions. Across all five datasets, paired comparisons yielded p-values well above the standard 0.05 threshold for statistical significance: p equals 1.0000 for Br35H, FigShare, and the four-class dataset, p equals 0.2891 for the fifteen-class dataset, and p equals 0.5488 for the thirty-class benchmark. On the four-class and thirty-class datasets, the ConvNeXt V2 anchor alone achieved 96.81 percent and 97.93 percent accuracy, respectively, compared to 96.75 percent and 97.73 percent for CAFM v4. Consequently, the authors explicitly noted that the empirical differences between CAFM v4 and its primary convolutional anchor must be characterized as numerical variations rather than statistically significant superiorities.

The study also highlighted several methodological constraints and calibration limitations that currently preclude clinical deployment. Formal evaluation of predicted probabilities demonstrated expected calibration errors ranging between 0.16 and 0.30 across the five datasets, indicating that despite high classification accuracy, the raw probability outputs remain uncalibrated and cannot be interpreted as direct clinical risk estimates. Furthermore, computational complexity is substantial, with the full 182.3-million-parameter model requiring 61.82 GFLOPs per forward pass, which, combined with multi-view test-time augmentation, results in single-view latencies between 20.5 and 53.6 milliseconds per image. The authors noted that in low-complexity settings, single-backbone architectures offer competitive accuracy with significantly reduced computational overhead.

Finally, the authors emphasized critical data-level limitations regarding dataset partitioning, clinical scope, and external generalization. A retrospective near-duplicate audit identified image-level leakage across partitions in several benchmark collections, including 719 near-duplicate pairs in Br35H, 1,009 in the four-class dataset, and 1,135 in the thirty-class dataset, while patient identifiers were either unavailable or unverified across most cohorts. Because each benchmark model was trained and evaluated independently on public image folders without multi-center or cross-scanner testing, the reported results do not demonstrate external or cross-institutional generalizability. The researchers concluded that extensive prospective validation on multicenter, patient-disjoint cohorts and post-hoc probability calibration will be essential before multi-branch deep learning systems can be considered for real-world neuro-oncological clinical decision support.

Subject of Research: Deep learning framework for brain tumor classification using multi-backbone residual fusion on MRI scans

Article Title: CAFM v4: a cascaded anchor-dominant residual fusion model for multi-scale brain tumor classification from magnetic resonance imaging

Article References: Kamineni, N., & Nadesh, R. K. (2026). CAFM v4: a cascaded anchor-dominant residual fusion model for multi-scale brain tumor classification from magnetic resonance imaging. Frontiers in Artificial Intelligence, 9, Article 1937548. https://doi.org/10.3389/frai.2026.1937548

Image Credits: AI Generated

DOI: 10.3389/frai.2026.1937548

Keywords: brain tumor classification, magnetic resonance imaging, deep learning, CAFM v4, ConvNeXt V2, Swin Transformer, EfficientFormerV2, residual fusion, medical imaging, neuro-oncology, cross-attention, stochastic weight averaging, test-time augmentation, computer-aided diagnosis, model calibration, benchmark evaluation