New Hashing Method Speeds Up Image and Text Search at Massive Scale

Searching a database of millions of images with a text query, or finding the right caption for a picture, is one of the defining challenges of modern multimedia systems. A research team led by Qian Zhang and Xin-Shun Xu at Shandong University, working with colleagues including Yu-Wei Zhan and Zhen-Duo Chen, has introduced a supervised cross-modal hashing method called Semantic-Enhanced Graph Regularization Hashing, or SEGRH, that addresses this challenge by rethinking how the supervisory signals guiding hash code learning are constructed. The work, published in the journal Data Science and Engineering on 2 November 2026, reports that the approach significantly outperforms previous state-of-the-art methods on three widely used benchmark datasets.

Cross-modal hashing exists because exhaustive comparison is impractical at scale. Hashing methods map images and texts into a short, shared binary space where semantic similarity is measured efficiently using Hamming distance between binary codes. This design keeps storage requirements low and retrieval speed high, which is why hashing has become a popular choice for large-scale systems. The core objective is to learn a separate hash function for each modality so that the resulting binary codes preserve semantic information within each modality while maintaining semantic relevance across modalities. In the SEGRH framework, images are converted into K-bit binary codes and texts into K-bit binary codes of the same length, so that an image query and a text database live in the same compact space.

Supervised cross-modal hashing methods exploit label information to make codes more discriminative, but the authors identify two persistent problems with how that supervision is typically used. Most current methods build supervision from cosine similarity between sample label vectors, which assesses only whether samples are relevant and does not measure the degree of semantic association between categories; this produces unreliable supervision signals that undermine the semantic consistency of hash codes. Alternative approaches use contrastive or triplet losses to pull similar cross-modal pairs together and push dissimilar ones apart, but these losses act only at the sample-pair level. They cannot impose global structural constraints on the continuous representation space before binarization, so although local pairwise relationships may be partially preserved, the overall semantic topology of the hash space lacks coherence, leading to inconsistencies in similarity measurement in the final Hamming space.

The team’s solution borrows a metric from natural language processing: positive pointwise mutual information, or PPMI. PPMI quantifies the strength of association between two elements based on how much more frequently they co-occur than expected by chance, and by retaining only positive values it naturally filters out spurious or coincidental correlations. The authors treat category labels analogously to words in a corpus. For a dataset of N multimodal samples they compute a co-occurrence matrix capturing how often category pairs occur together, estimate joint probabilities from the observed co-occurrence frequencies, and then derive a PPMI matrix in which each entry quantifies the association strength between two categories. This transforms raw label co-occurrence frequencies into fine-grained, continuous semantic association scores that go beyond binary relevance judgments, providing more precise supervisory signals for hash code learning.

Those association scores are then embedded into the geometry of the hash space through graph Laplacian regularization. The PPMI matrix is applied to compute a semantic similarity matrix across samples, which is normalized by label counts to avoid scale bias from multi-label samples. A degree matrix and Laplacian matrix are derived from this similarity structure, and a semantic preservation loss, expressed as the trace of a matrix product involving the continuous representations and the Laplacian, constrains the distribution of the codes. Through this mechanism the method forces semantically similar samples to cluster together in the hash space, ensuring that the distribution of hash codes aligns with the semantic structure globally rather than only pairwise. The learned continuous representations are finally binarized into hash codes through the sign function.

The second core component of SEGRH is contrastive cross-modal fusion. The method uses the pre-trained CLIP model as the backbone to extract modality-specific features, unfreezing the last layers of both encoders so they can adapt to the retrieval datasets. A cross-modal alignment loss aligns the extracted image and text representations in a shared semantic space. The concatenated features are then fed into a multimodal Transformer, whose self-attention mechanism explicitly models cross-modal associative patterns, and a fusion loss reinforces the interaction between visual and textual features. Two multi-layer perceptrons with tanh activation map the fused representations into approximate hash codes, and a contrastive hashing loss optimizes the consistency of these approximate codes across modalities. The authors argue that enforcing contrastive constraints at every step ensures the semantic alignment achieved by CLIP is not lost during multimodal fusion and binarization, maintaining a consistent similarity structure throughout the encoding pipeline. The full objective combines the four loss terms with hyperparameters controlling each contribution.

The experimental evaluation used three real-world datasets with the same splits as previous cross-modal retrieval methods. MIRFlickr comprises 25,000 image-text pairs annotated with as many as 24 semantic labels, of which 20,015 valid samples remain after omitting pairs with fewer than 20 labels; 2,000 pairs serve as queries, 18,015 form the retrieval database, and 5,000 are sampled for training. NUS-WIDE is a large-scale dataset of 269,648 images labeled with 81 concepts, reduced to the top 10 most common categories, yielding 186,577 image-text pairs with 2,000 queries, a database of 184,577 pairs, and 5,000 training pairs. MS COCO contains 123,287 image-caption pairs spanning 80 object categories, with 2,000 query pairs, a 121,287-pair database, and 5,000 training pairs. Feature extraction used CLIP based on the ViT-B/32 architecture, the multimodal Transformer has two encoder layers with four attention heads, a hidden dimension of 1024 and an intermediate feed-forward dimension of 2048, and hyperparameters were tuned per dataset through a fine-grained grid search.

Performance was assessed with mAP@50, which computes the average precision for each query based on only the top 50 retrieved results, and with precision of top-N curves, on two tasks: image-to-text retrieval and text-to-image retrieval, with similarity measured by Hamming distance. The mAP@50 comparison and ablation results were averaged over five independent runs. The results show that SEGRH significantly outperforms all baseline methods at every tested hash length across the three benchmark datasets, against a field of 13 state-of-the-art competitors including four unsupervised methods and nine supervised ones. On NUS-WIDE, SEGRH surpasses BiLGSEH, the second-best method, by an average of 4% in the image-to-text task and 7.7% in the text-to-image task. To ensure a fair evaluation, the team also uniformly replaced the backbone networks of the baseline methods with CLIP; despite the resulting performance boost across the baselines, SEGRH maintained its leading performance, outperforming nine baseline methods in both tasks across the three datasets. Precision of top-N curves showed SEGRH consistently outperforming both supervised and unsupervised approaches as the number of retrieved samples increased.

Ablation studies on MIRFlickr probed why the method works. Replacing the Laplacian regularization with a mean squared loss produced the lowest performance among all variants, suggesting the regularization is essential for guiding hash code optimization and yielding more discriminative representations. Swapping co-occurrence statistics and PPMI for cosine supervision between label vectors also underperformed the full method, indicating that PPMI offers better supervision. Using frozen CLIP encoders without fine-tuning the last residual attention blocks hurt performance as well, showing that fine-tuning helps CLIP adapt to cross-modal retrieval datasets. Removing any of the four individual loss terms degraded retrieval accuracy in separate experiments, indicating that each plays a crucial role. Hyperparameter sensitivity analysis found that the alignment weight degrades performance above 0.1, the fusion weight is optimal within a range from 0.1 to 1, the contrastive hashing weight peaks near 1 with sharp drops otherwise, and the semantic preservation weight performs well below 1, peaks at 1, and declines sharply above it, with trends roughly consistent across datasets.

The paper also reports practical advantages beyond accuracy. A convergence analysis found that retrieval accuracy rises rapidly in early training stages and stabilizes after several epochs, with MIRFlickr converging faster and more steadily than NUS-WIDE and MS COCO, likely because of its simpler semantic structure, while the two larger datasets with higher label density and semantic complexity require more iterations. A training efficiency comparison against BiLGSEH on pre-extracted features showed that SEGRH consistently requires less training time across all hash code lengths and datasets, an improvement the authors attribute to a streamlined formulation that reduces redundant computations during optimization, making the method more suitable for large-scale applications where computational cost is a critical concern. Qualitative visualization with 64-bit hash codes showed the method retrieving relevant results across modalities, including surfacing images of people surfing when queried with the phrase about a man riding a wave on a surfboard. The authors state that data will be made available on request, and they hope the work offers a new perspective for designing supervised cross-modal hashing algorithms.

Subject of Research: A supervised cross-modal hashing method that uses positive pointwise mutual information and graph Laplacian regularization to improve large-scale image-text retrieval.

Article Title: Semantic-Enhanced Graph Regularization Hashing for Large-Scale Cross-Modal Retrieval

Article References: Zhang, Q., He, S.-H., Jiang, H.-H., Liu, X.-Q., Zhan, Y.-W., Chen, Z.-D., Luo, X., & Xu, X.-S. (2026). Semantic-Enhanced Graph Regularization Hashing for Large-Scale Cross-Modal Retrieval. Data Science and Engineering. https://doi.org/10.1007/s41019-026-00366-6

Image Credits: AI Generated

DOI: 10.1007/s41019-026-00366-6

Keywords: cross-modal retrieval, hashing, CLIP, PPMI, graph Laplacian regularization, multimodal Transformer, Hamming distance, MIRFlickr, NUS-WIDE, MS COCO, semantic similarity, large-scale search