Early Modality Classifier Cuts Multimodal Retrieval Latency by Nearly Half

Standard multimodal retrieval-augmented generation systems routinely execute separate searches across both text passages and visual collections for every incoming user request, retrieving image and text embeddings even when a query can be resolved entirely through textual information alone. In an evaluation conducted across a curated collection of natural language queries, researchers demonstrated that inserting a lightweight neural classifier before visual retrieval can reduce end-to-end pipeline latency from 118.97 milliseconds to 65.61 milliseconds per query, representing an overall processing time reduction of 44.85 percent. Rather than treating visual search as an obligatory step for every query, the framework establishes a supervised decision boundary that determines whether image retrieval is genuinely warranted. Operating on an experimental benchmark of 265 labeled queries matched against an image collection of 5,000 indexed photographs and their corresponding textual captions, the proposed pre-retrieval selection model achieved an accuracy of 94.34 percent on a held-out test split of 53 queries, incurring one unnecessary visual retrieval and two missed visual retrievals while demonstrating that multimodal systems can bypass computationally demanding image encoders and index searches for purely textual inquiries.

The engineering motivation behind modality selection stems from the operational architecture of contemporary retrieval-augmented systems, which increasingly integrate both dense textual retrievers and vision-language models to ground generated responses in external knowledge. While incorporating multimodal context into response generation assists automated systems in representing real-world environments more accurately, prevailing retrieval pipelines handle incoming requests uniformly by issuing parallel queries against text databases and image indexes. For queries that merely seek conceptual explanations or factual summaries, executing visual feature extraction and nearest-neighbor vector scans consumes substantial processor cycles and imposes substantial latency overhead without contributing useful information to downstream generation components. Prior multimodal machine learning investigations have primarily targeted cross-modal alignment, shared representation spaces, and cross-attention fusion mechanisms, giving comparatively little consideration to the preliminary operational question of whether visual assets are actually needed for a given request. Existing methods that attempt to manage multimodal activation typically rely on static heuristic rules or raw similarity thresholds that fail to adapt across heterogeneous query structures.

To investigate whether modality routing can be systematized without imposing extensive computational overhead, the study formalizes modality selection as a supervised binary classification problem positioned within an information retrieval workflow. The researchers designed an operational framework comprising four consecutive modules: query processing, embedding generation, index retrieval, and modality selection. In this architecture, textual retrieval is always performed first as an invariant baseline operation, utilizing Sentence-BERT to convert natural language queries into 384-dimensional dense vectors that are matched against pre-indexed passage embeddings using the FAISS approximate nearest neighbor library to produce a mean textual relevance score. The routing architecture then branches depending on where the modality determination is staged. Under a post-retrieval configuration, the system processes every query through a 512-dimensional CLIP ViT-B/32 text encoder and searches an image index before classification occurs, using the resulting visual similarity score to decide whether to transmit retrieved visual assets to downstream components. Under the pre-retrieval configuration, the classifier acts immediately after initial text retrieval, evaluating query-level attributes and text-retrieval metrics to determine whether the entire visual retrieval branch should be skipped outright.

The feature vector constructed for the classification model reflects both retrieval confidence indicators and surface query characteristics. In the full six-dimensional post-retrieval setup, the feature representation incorporates the text relevance score produced by Sentence-BERT, the visual relevance score generated by CLIP image retrieval, the mathematical difference between visual and textual relevance scores, the character or token length of the query, a visual intent indicator flag derived from the presence of explicit lexical markers such as show, image, picture, scene, photograph, or diagram, and an interaction term combining the visual intent score with the image relevance score. Because extracting the image relevance feature inevitably requires running the CLIP encoder and performing an index lookup across visual embeddings, that six-feature variant cannot prevent the visual search from occurring. To construct a genuine pre-retrieval routing mechanism, the authors formulated an alternative four-dimensional feature set that entirely excludes visual search outputs. This pre-retrieval feature set retains the text relevance score, query length, and the lexical visual-intent indicator, while replacing the visual interaction term with an interaction calculated between the visual intent indicator and the text relevance score, ensuring all required inputs are accessible before querying visual indices.

The classification backbone responsible for deciding modality routing consists of a compact Multi-Layer Perceptron implemented in PyTorch, comprising two fully connected layers with rectified linear unit activations, a dropout regularization layer with a probability parameter set to 0.2, and a terminal sigmoid activation function that outputs a calibrated probability reflecting whether a query requires visual augmentation. Across both feature configurations, the neural network contains approximately 1,500 trainable parameters, rendering its runtime execution cost negligible when contrasted with the dense vector encoders and approximate nearest neighbor searches governing the underlying retrieval pipeline. Training was executed using the Adam optimization algorithm under binary cross-entropy loss with a weight decay regularizer of 10⁻⁴ over 150 epochs, employing a fixed random seed of 42 across all experimental runs to maintain numerical reproducibility. The continuous output score generated by the network is mapped to a binary routing decision using a predetermined decision threshold fixed at 0.47, establishing an operational point that governs whether the visual branch is triggered or suppressed.

Evaluation of the modality selection framework was performed on an experimental dataset constructed from a 5,000-image subset of the Flickr30k corpus. In this benchmark configuration, each individual image was indexed separately for visual retrieval, while the associated human-authored captions were stored as standalone textual documents within the text retrieval index. The test collection encompassed 265 natural language queries developed to reflect practical user requests, partitioned initially into 120 visual queries, 120 text-only queries, and 25 queries classified as ambiguous during authoring. Ground-truth binary labels denoting whether answering each query appeared to call for visual information were produced independently by two external computer science graduate students who had no involvement in authoring queries, designing features, or constructing the software pipeline. The annotators demonstrated substantial inter-rater reliability, achieving complete agreement on 258 out of the 265 queries, which corresponded to a Cohen’s kappa coefficient of 0.947 with a 95 percent confidence interval spanning 0.907 to 0.985. The seven instances of annotator disagreement were concentrated almost entirely within the ambiguous category, accounting for five discrepancies, and all disagreements occurred within the training split, leaving unanimous agreement across the held-out test set.

The dataset was partitioned using an 80/20 stratified split that preserved relative class balance, assigning 212 queries to model training and 53 queries to the held-out test evaluation partition, which comprised 26 visual queries and 27 text-only queries. Within the 53-query test partition, the proposed pre-retrieval-only Multi-Layer Perceptron achieved an overall classification accuracy of 94.34 percent, corresponding to a 95 percent Wilson score confidence interval ranging from 84.6 percent to 98.1 percent. This model produced a macro-averaged F1 score of 0.9433, successfully identifying 24 out of 26 visual queries for a visual recall of 92.3 percent, while incorrectly routing only one text-only query to the visual retrieval pipeline for a visual precision of 96.0 percent. The post-retrieval Multi-Layer Perceptron variant using all six features yielded an identical test accuracy of 94.34 percent and a macro F1 score of 0.9431, but it missed three visual queries, producing a lower visual recall of 88.5 percent and a visual precision of 100 percent.

The research compared both neural selector variants against ten baseline routing strategies, including three heuristic methods and seven supervised learning algorithms trained on identical data splits. A naive multimodal baseline that retrieved both text and images for every query achieved an accuracy of 49.06 percent with a macro F1 score of 0.3291, making zero missed visual retrievals but generating 27 unnecessary image searches out of 53 test requests. A keyword-based heuristic relying exclusively on lexical intent terms attained 88.68 percent accuracy, whereas a semantic-similarity baseline comparing raw retrieval scores collapsed to an accuracy of 39.62 percent by failing to retrieve visual assets for all 26 visual queries. Across the learned baselines utilizing the full six-dimensional feature representation, logistic regression, an RBF-kernel support vector machine, a decision tree, a random forest, and an XGBoost model converged on identical decision boundaries, each recording 94.34 percent accuracy and committing the exact same three routing errors as the post-retrieval neural network. A text-score-only classifier relying solely on the Sentence-BERT relevance metric achieved an accuracy of 92.45 percent.

Crucially, an isolated query-only baseline classifier operating strictly on surface lexical features, specifically query token length and keyword intent markers without access to any text or image retrieval scores, achieved 98.11 percent accuracy on the held-out test set, committing only a single classification error across all 53 queries. This query-only model demonstrated that in the evaluated query collection, a substantial portion of the label variance could be extracted directly from query surface structure without performing intermediate retrieval scoring. Because the query-only model recorded one misclassification while the pre-retrieval neural selector recorded three, the two classifiers differed on at most four test instances, meaning that McNemar’s exact test produces a two-sided p-value well above 0.5. Consequently, the authors emphasized that the primary technical contribution of their work rests on the structural design of the pre-retrieval routing mechanism and its demonstrable reduction in pipeline latency, rather than on superior statistical accuracy over simpler linear or surface-level classifiers.

To contextualize operational risk within production environments, the study analyzed model performance under a cost-sensitive metric defined as the number of unnecessary visual retrievals plus three times the number of missed visual retrievals. This illustrative threefold penalty models practical engineering scenarios where withholding necessary imagery from a generation model degrades response utility far more severely than issuing an unneeded visual query. Under this formulation, the query-only classifier attained the lowest overall cost score of 3.0, combining perfect visual precision of 1.000 with 96.2 percent visual recall. Among retrieval-score-dependent architectures, the pre-retrieval Multi-Layer Perceptron registered the lowest cost score of 7.0 by virtue of its single unnecessary retrieval and two missed retrievals, outperforming the post-retrieval network and classical supervised baselines, which each registered cost scores of 9.0 due to three missed retrievals. Conversely, the naive multimodal pipeline accumulated a cost score of 27.0 from 27 unnecessary retrievals, while the degenerate semantic-similarity heuristic registered a severe cost score of 84.0 by missing every visual query.

An ablation study conducted using 5-fold cross-validation across the full 265-query dataset examined the relative contribution of individual input features within the post-retrieval model architecture. In this cross-validation analysis, the full-feature Multi-Layer Perceptron established a baseline mean accuracy of 95.85 percent with a cross-fold population standard deviation of 2.77 percent, alongside a mean macro F1 score of 95.83 percent with a 2.79 percent standard deviation. Removing the text relevance score produced the largest measurable degradation, reducing mean cross-validation accuracy by 1.90 percentage points to 93.96 percent. Ablating the image relevance score and the relevance score difference each diminished accuracy by 1.10 percentage points, while omitting query length resulted in a 0.75 percentage point decrease. In contrast, removing either the visual intent flag or its interaction term produced a net change of zero across the five folds, caused by symmetric 1.9 percentage point fluctuations on individual folds that canceled each other out. Because all observed ablation differences remained within the 2.77 percentage point variation observed across cross-validation folds, the authors characterized these individual feature contributions as directional indicators rather than individually decisive factors.

Direct latency benchmarking provided empirical validation for the efficiency advantages of pre-retrieval routing. Instrumentation was conducted on an Intel Core i7-1255U central processing unit featuring 12 logical cores and 16.8 gigabytes of memory, executing in Python 3.12 under Windows 11 with single-threaded sequential execution at batch size 1 without hardware acceleration. Latency metrics were averaged over 20 repeated passes across the 53-query test set following two discarded warm-up cycles. The baseline naive pipeline required 118.97 ± 3.40 milliseconds per query. The post-retrieval selector lowered processing time to 101.46 ± 2.94 milliseconds, achieving a 14.72 percent latency reduction strictly by avoiding downstream image decoding, loading, and re-encoding steps, which had an empirical average cost of 32.0 ± 0.9 milliseconds per query. In contrast, the pre-retrieval selector lowered total execution time to 65.61 ± 1.98 milliseconds per query, securing a 44.85 percent reduction by skipping the CLIP text encoding pass, the FAISS image index search, and all subsequent visual handling steps whenever queries were identified as text-only.

The authors outlined several methodological constraints that define the scope of their reported findings. First, the evaluation benchmark was relatively small, consisting of 265 total queries and a 53-query held-out split where each misclassification shifted observed accuracy by approximately 1.89 percentage points, yielding wide Wilson confidence intervals. Second, all queries were authored by the researchers who formulated the feature sets, meaning that broader benchmark validation against external user queries is necessary to verify distributional robustness. Third, because the text corpus consisted exclusively of Flickr30k image descriptions, text-only queries structurally lacked matching conceptual documents, meaning that text relevance scores partly reflected image corpus alignment rather than pure textual answerability. Finally, the authors noted that their experimental protocol evaluated classification routing decisions against human judgments rather than measuring the downstream factual correctness, response quality, or completeness of text generated by an integrated language model, leaving the ultimate impact of bypassed retrieval on end-to-end generation quality as an essential subject for subsequent research.

Subject of Research: Learning-based modality selection for efficient multimodal retrieval-augmented generation systems

Article Title: Learning-based modality selection for efficient multimodal retrieval

Article References: Kannekanti, R. C., & Malghan, R. L. (2026). Learning-based modality selection for efficient multimodal retrieval. Discover Computing, 29(1), Article 783. https://doi.org/10.1007/s10791-026-10733-6

Image Credits: AI Generated

DOI: 10.1007/s10791-026-10733-6

Keywords: multimodal retrieval, retrieval-augmented generation, modality selection, pipeline latency, computational efficiency, Sentence-BERT, CLIP embeddings, FAISS indexing, multilayer perceptron, classification accuracy, query routing, visual intent detection, Flickr30k dataset, cost-sensitive evaluation, lexical intent markers, feature ablation, cross-validation