Survey Maps a Roadmap for Cross-Modal Search in India’s Languages

The country recognizes more than twenty official languages and hosts an even greater number of dialects, yet established retrieval platforms continue to favor high-resource languages such as English and Hindi.

The paper, authored by Revati Wahul and colleagues, reviews more than a decade of research on cross-modal retrieval, the paradigm in which a user queries in one modality, such as text, and receives results in another, such as an image or an audio clip. The team searched Scopus, IEEE Xplore and Google Scholar for publications from 2016 to 2026, using keywords including cross-modal retrieval, multilingual information access, Indic languages, retrieval-augmented generation and low-resource NLP. From an initial pool of more than 400 results, quality checks and duplicate removal left 286 core papers that form the survey’s comparative baseline.

The survey’s central finding is that the components needed for inclusive multilingual search largely exist, but in fragments. Neural machine translation, transformer-based embeddings, graph-based fusion architectures and retrieval-augmented generation have each advanced rapidly, yet their integration for Indian regional languages remains underexplored. Existing systems struggle with script normalization, noise in low-resource corpora, and translation that must preserve cultural and contextual nuance. Code-mixing, the blending of languages within a single sentence that is common in everyday Indian speech, compounds the problem, as does the scarcity of labeled multimodal corpora for most regional languages.

The authors illustrate the state of the art with examples drawn from adjacent research areas. One reviewed system, a Vision-Language Transformer Combination Search framework, fuses embeddings from three vision transformers and three language models to classify memes as misogynistic or non-misogynistic, achieving an F1-score of 0.7924 on the Gasparini dataset and 0.728 on SemEval-2020. The survey notes that such architectures demonstrate strong semantic alignment between text and image, but remain confined to binary classification, leaving their potential as indexing and ranking engines for retrieval untapped.

Other reviewed studies underscore both the promise and the fragility of current approaches. A hybrid Bengali cyberbullying detector combining Banglish normalization, TF-IDF feature extraction and Instance Hardness Threshold undersampling reached 98.82 percent accuracy with a 98.88 percent F1-score. A Romanized Urdu classifier built on mBERT topped out at 0.799, which the survey cites as evidence of critical failures in low-resource, domain-tailored models. In crisis informatics, the CReMa model showed that Gujarati and Hindi, classed as low-resource, still trailed significantly in retrieval performance at 0.61 and 0.69 respectively. The survey is careful to note that these results come from different tasks, datasets, metrics and hardware, so they are not directly comparable, and its revised comparative figures present qualitative coverage only.

That methodological caution runs through the paper. Because the underlying studies address different problems, from classification to ranking to question answering, the authors explicitly decline to rank techniques by performance or efficiency. Their coverage maps record only whether a technique or modality is used at each pipeline stage. Those maps reveal a striking pattern: text dominates every stage of the surveyed pipelines, while audio and video appear mainly at embedding and output layers, and speech and logs are rarely addressed. External knowledge infusion, transliteration-aware encoders and script-agnostic Unicode pipelines appear in only a minority of the architectures reviewed, and vision-centric content-based image retrieval systems include none of the Indic-relevant linguistic features at all.

From these gaps the paper derives its position statement: TriModalFusion-LM, a conceptual architecture for unified cross-modal retrieval in Indian languages. The proposed pipeline begins with a heterogeneous input ingestion layer that applies Indic NLP normalization to mitigate orthographic variance, script-level ambiguity and dialect divergence. An IndicTrans translation module then maps bidirectionally across Indian regional languages, with an optional code-switching restoration stage whose value is to be judged by downstream retrieval quality rather than translation metrics alone. An LLM-driven text optimization stage follows, using low-temperature decoding and task- and domain-conditioned prompts, and deliberately avoiding chain-of-thought prompting, which prior work cited in the survey links to degraded translation quality and increased hallucination for non-English-centric language pairs.

Downstream, the optimized text is embedded with Sentence Transformers encoders, TimeSformer encoders produce spatio-temporal video embeddings, and a wav2vec2-based acoustic encoder captures phonetic and prosodic audio features. A multimodal graph convolutional fusion layer models inter-modal dependencies and relational hierarchy to produce a unified cross-modal representation, which is indexed in an approximate nearest-neighbour store. Candidate items are retrieved by dense similarity search and passed to a retrieval-augmented generation module that grounds responses in retrieved evidence, with IndicTrans back-translation proposed as a semantic consistency check before delivery. The final response is returned as text in the query language and, through gTTS or indic-tts synthesis, as speech in the user’s regional language, an accessibility feature aimed at users with visual impairment or limited literacy.

The authors are unambiguous that none of this has been built. TriModalFusion-LM is not implemented or empirically evaluated in the work; no experiments were conducted, and no retrieval-quality metrics such as Recall, NDCG or MRR are reported for the system. The computational cost of the pipeline, which combines several resource-intensive components, remains unquantified, despite low-resource and on-device deployment being a stated motivation. The behavior of the components once jointly integrated, including potential compounding errors across modules, has not been studied and may differ from their individually reported performance. The survey itself also has limits: the search was restricted to English-language publications across three databases, and coverage across India’s 22 scheduled languages is uneven, with most studies addressing Hindi, Bengali or Tamil and many low-resource languages represented by few or no retrieval studies.

What the paper delivers, then, is a structured diagnosis and a testable design rather than a working system. The authors outline an evaluation protocol that includes retrieval with and without code-switching restoration, a sweep of decoding temperature and prompt variants for the LLM optimization stage, and a hallucination analysis on non-English-centric Indic language pairs. They also flag a broader constraint: general-purpose, labeled multimodal and code-mixed corpora remain scarce for the majority of Indian regional languages, including Bengali, Tamil, Marathi, Urdu and Assamese, with most existing resources concentrated in narrow domains or in Hindi alone. Code-mixed and Romanized queries such as Hinglish and Tanglish compound the challenge, since language identification, script normalization and cross-lingual alignment for such inputs remain inconsistently addressed, with most solutions validated on a single code-mixed pair rather than across the broader range of Indian code-mixing patterns. Whether the proposed fusion of translation, multimodal embedding and retrieval-augmented generation can deliver linguistically equitable search at scale is now an empirical question the survey has, at least, made precise.

Subject of Research: A systematic survey and position paper on scalable cross-modal retrieval for multilingual information access in Indian languages, proposing a conceptual architecture called TriModalFusion-LM.

Article Title: A systematic survey of scalable cross modal retrieval for inclusive multilingual information access in Indian languages

Article References: Wahul, R., Kale, V. V., Parlewar, P., Shinde, A., Dharwadkar, S. N., Bhole, C., Jagtap, V., Gajare, M. P., Pole, G. S., Bhardwaj, R., Bhavsar, S. S., & Gawande, S. H. (2026). A systematic survey of scalable cross modal retrieval for inclusive multilingual information access in Indian languages. Discover Computing, 29(1), Article 813. https://doi.org/10.1007/s10791-026-10771-0

Image Credits: AI Generated

DOI: 10.1007/s10791-026-10771-0

Keywords: cross-modal retrieval, multilingual information access, Indian languages, low-resource NLP, TriModalFusion-LM, systematic survey, multimodal embeddings, retrieval-augmented generation, neural machine translation, code-mixing, Indic languages, position paper