Hybrid AI Model Retrieves Medical Images Even When Queries Are Blurred or Noisy

Medical image archives are growing faster than clinicians can search them, and a new open-access study argues that the answer lies in a hybrid artificial intelligence architecture that blends two very different ways of seeing. Researchers report a content-based medical image retrieval (CBMIR) system that combines a convolutional neural network with a lightweight transformer encoder, producing compact 128-dimensional image embeddings that remain usable even when query images are blurred, noisy, cropped, or partially obscured. The work, published in Discover Life, was evaluated on four public benchmarks spanning chest radiographs, heterogeneous radiology collections, and gastrointestinal endoscopy images.

The core problem the authors tackle is practical rather than theoretical. Content-based retrieval lets clinicians find previous cases that look visually and semantically similar to a query image, bypassing reliance on DICOM tags or keyword searches. Yet real-world deployment is hindered by multiple imaging modalities, scarce labeled data, and vulnerability to everyday artifacts such as cropping and blurring. Earlier deep learning systems achieved strong accuracy on clean, curated datasets, but robustness analysis under realistic degradation has often been superficial, according to the study.

The proposed architecture works in three stages. First, an ImageNet-pretrained ResNet-50 backbone extracts high-resolution local features from the conv5 stage of the network, capturing textures, edges, lesion boundaries, and organ borders. These spatial features are reshaped into a sequence of 64 tokens and passed to a transformer encoder with just two layers and eight attention heads, which models long-range anatomical relationships such as lesion-to-organ context. Finally, a learnable gated fusion module dynamically interpolates between the local and global signals for each embedding dimension, and a projection head maps the fused representation to a 128-dimensional, L2-normalized vector in which cosine similarity directly measures image resemblance.

Training deliberately simulates the corruptions that clinical images accumulate in practice. The augmentation pipeline includes additive Gaussian noise, salt-and-pepper noise, Gaussian blurring, sharpening filters, gamma corrections, random rotations of up to ten degrees, scaling, and center-cropping. By forcing the network to learn representations anchored in stable anatomical structure rather than transient pixel signatures, the authors aimed to close the gap between benchmark conditions and deployment conditions. Evaluation was then carried out under both clean and intentionally degraded queries, using Precision@K, Recall@K, and mean average precision as unified metrics.

The benchmarks were chosen to span modality and difficulty. ImageCLEFmed 2009 contributes 12,677 labeled radiographs across 193 IRMA-coded categories, while ImageCLEFmed 2007 adds roughly 12,000 grayscale X-rays in 116 categories, both marked by substantial intra-class variability and strong visual overlap between related codes. The COVID-19 Radiography Database supplies more than 21,000 chest X-rays across four diagnostic classes with pronounced imbalance, and the extended Kvasir GI v2 dataset provides 8,000 endoscopic images across eight classes, ranging from normal anatomical landmarks to polyps, ulcerative colitis, and esophagitis.

An ablation study on the ImageCLEFmed 2009 clean test split quantifies why each component matters. A CNN-only variant reached a Precision@1 of 0.83, while a transformer-only configuration performed worst at 0.75, with mean average precision at ten of 0.70, reflecting the absence of the localized inductive biases that convolutions provide. Merging both streams through static concatenation lifted Precision@1 to 0.86, and replacing that static fusion with the dynamic gated module delivered an additional 4.0 percent absolute improvement in both Precision@1 and mAP@10, which the authors attribute to context-dependent re-weighting of local versus global evidence.

The robustness results are the study’s most striking claim. On the ImageCLEFmed 2009 benchmark under systematic query corruptions, Precision@1 held at 0.89 for noise and blur perturbations, with mAP@10 remaining between 0.83 and 0.85 for moderate, low-severity degradations, nearly matching the clean-query environment. Even under severe degradations such as heavy cropping or partial structural occlusion, where much of the spatial content is lost, the model still produced Precision@1 of at least 0.80 and 0.79, respectively. The authors link this stability directly to the hybrid design: when local details are corrupted, the gated fusion module down-weights the unreliable local pathway and leans on global anatomical layout instead.

Scalability experiments varied the size of the retrieval index across all four datasets. Precision@1 increased monotonically with gallery size everywhere, with no evidence of hubness or nearest-neighbor collapse, and saturation arrived earlier for ImageCLEFmed 2009 than for the more heterogeneous 2007 collection. Two-dimensional PCA projections of the learned embeddings showed coherent structure: a compact cluster of normal chest radiographs separate from a broader pathological lobe in the COVID-19 data, texture-driven lesion clusters in Kvasir, and dense islands corresponding to frequent IRMA codes in the ImageCLEFmed sets, with residual overlap between clinically similar categories mirroring known visual ambiguity.

The authors are explicit about limitations. The method handles only 2D images, evaluation relies on label-exact metrics rather than radiologist-centered relevance judgments, and retrieval uses a straightforward cosine index rather than advanced approximate nearest-neighbor or hashing structures. Comparisons with prior systems such as transformer-based hashing and large-scale graph-indexed retrieval are presented as an informative literature overview rather than a controlled head-to-head benchmark, since datasets, splits, and relevance criteria differ. Future work outlined in the paper includes multi-label relevance criteria, integration with foundation models such as BiomedCLIP through distillation or re-ranking, extension to 3D retrieval, and deployment as an open-source PACS plugin to study relevance feedback with working radiologists.

For now, the study positions the hybrid encoder as a task-specific alternative that may complement both ViT-based hashing pipelines and general-purpose foundation models in medical image search. Its central message is that local detail and global context are complementary, not competing, and that a small gated module deciding how much to trust each stream under degraded conditions can make the difference between a retrieval system that works only on pristine benchmarks and one that survives contact with the noisy, artifact-laden images of real clinical workflows.

Subject of Research: A hybrid CNN–transformer architecture for retrieving visually similar medical images under real-world image degradation

Article Title: Hybrid CNN–transformer architecture for content based medical image retrieval

Article References: Bhatt, N., Wary, A., & Kumari, R. (2026). Hybrid CNN–transformer architecture for content based medical image retrieval. Discover Life, 56(1), Article 37. https://doi.org/10.1007/s11084-026-09757-1

Image Credits: AI Generated

DOI: 10.1007/s11084-026-09757-1

Keywords: content-based image retrieval, medical imaging, CNN, transformer, ResNet-50, gated fusion, robustness, ImageCLEFmed, COVID-19 Radiography Database, Kvasir endoscopy, embeddings, data augmentation