Unsupervised Network Produces Mechanistic Saliency Maps for Medical Image Classifiers

Interpreting why a convolutional neural network reaches a diagnostic conclusion is a persistent challenge in medical imaging, and a new framework called SalientNet now proposes an unsupervised way to generate saliency maps that its developers describe as more mechanistically grounded and reliable than established visualization methods. The work, published as an Original Article in the International Journal of Computer Assisted Radiology and Surgery, addresses well-known weaknesses of standard saliency techniques, including low sensitivity to input perturbations, a lack of causal grounding, and limited mechanistic insight into how a model arrives at its decisions.

SalientNet is designed to be data- and architecture-agnostic, meaning it can be applied to different CNN classifiers without retraining on ground-truth saliency annotations. The framework consists of two jointly trained components. The first is a CNN examiner that uses a novel variable channel convolution module to extract architecture-independent information from a network’s intermediate feature maps. The second is an attribution map generator, built from two coupled U-Nets, which combines this extracted information with the original input image to produce a saliency map highlighting the regions that matter for the classifier’s decision.

The training signal comes from an Interpretation Loss rather than human-labeled explanations. This loss enforces that the generated attribution map retains just enough information for a frozen classifier to reproduce its original prediction within a controlled tolerance, referred to as a dissimilarity temperature. According to the authors, this constraint prevents both trivial masks that keep everything and overly aggressive masks that discard too much, pushing the explanations toward the minimal set of image regions that genuinely support the prediction.

To evaluate the approach, the team tested SalientNet on three medical imaging datasets spanning different modalities and body regions: BreakHis, a histopathology collection; Brain MRI ND5; and FracAtlas, a musculoskeletal X-ray dataset. Three widely used classifiers—ResNet-18, MobileNetV3-Small, and EfficientNet-B0—served as the interpreted networks, giving the evaluation coverage across multiple dataset-architecture combinations.

The reported results show SalientNet outperforming popular saliency-based benchmarks, including Grad-CAM, Grad-CAM++, and LIME. Across nearly all dataset-architecture combinations, the framework achieved substantially higher faithfulness, measured as Win %, and lower prediction Drop %, while keeping spurious confidence increases, measured as Inc %, comparably low. These metrics probe whether an explanation truly reflects the regions a classifier relies on: a faithful saliency map should, when restricted to highlighted areas, preserve the model’s prediction, and when those areas are removed, the prediction confidence should drop rather than rise.

Beyond benchmark metrics, the authors examined whether SalientNet’s explanations align with expert annotations. On FracAtlas, the framework’s explanations showed strong spatial correlation with fracture regions annotated by medical practitioners, with a mean ground-truth active fraction of up to 0.96 for ResNet-18. This suggests that, at least for fracture detection on X-rays, the unsupervised maps concentrate on clinically relevant anatomy without ever being trained on those annotations.

The study also includes a mechanistic analysis of the interpreted networks themselves. It found that nearly all CNN layers contribute meaningfully to SalientNet’s explanations, which the authors present as support for the interpretive fidelity of the framework. Rather than attributing a model’s behavior to a small set of late-stage layers, the analysis indicates that information distributed throughout the network shapes the final saliency output.

The researchers position the framework as a step toward more trustworthy AI-assisted medical diagnosis, offering local validation of CNN decisions in healthcare settings where both global and local checks are considered essential for building trust. They emphasize that the method requires no ground-truth saliency annotations for training, which reduces the burden of producing expert-labeled explanation data. The implementation has been made publicly available on GitHub, allowing other groups to apply or extend the approach. As with any interpretability method, the reported gains are specific to the tested datasets, architectures, and evaluation metrics, and adoption in clinical workflows would depend on further validation beyond this study’s scope.

Subject of Research: An unsupervised neural network framework for generating reliable saliency maps explaining CNN decisions in medical image classification.

Article Title: SalientNet: mechanistic and reliable saliency maps for medical image classification via unsupervised learning

Article References: Sekar, S., Akash Guna, R. T., González Ballester, M. A., & Sikha, O. K. (2026). SalientNet: mechanistic and reliable saliency maps for medical image classification via unsupervised learning. International Journal of Computer Assisted Radiology and Surgery. https://doi.org/10.1007/s11548-026-03812-y

Image Credits: AI Generated

DOI: 10.1007/s11548-026-03812-y

Keywords: SalientNet, saliency maps, explainable AI, convolutional neural networks, medical imaging, unsupervised learning, Grad-CAM, histopathology, brain MRI, X-ray fracture detection, interpretability, healthcare AI