Multimodal Retrieval Framework Enhances Technical Reasoning and Accuracy in Architectural Design

Architectural engineering and design workflows require synthesizing highly heterogeneous data sources, ranging from prescriptive building codes and complex structural diagrams to multi-level performance tables and annotated construction drawings. Traditional information retrieval pipelines and standard text-based retrieval-augmented generation systems frequently fail when applied to these workflows because converting visual and tabular assets into flat text strips away essential geometric arrangements, spatial relationships, and hierarchical headers. To resolve these operational shortcomings, researchers Xianchuan Meng and Ziyu Tong introduced a domain-tailored framework designated as Multimodal Retrieval-Augmented Generation, or M-RAG, detailed in a study published in Architectural Intelligence. Their framework directly integrates multimodal large language models with a unified knowledge base through a shared semantic embedding space, dual-mode table decomposition, and adaptive query-processing pipelines. By combining dedicated encoders for text and imagery with rigorous relevance filtering, the system establishes a traceable cross-modal retrieval and generation workflow capable of answering complex architectural queries while significantly curbing technical hallucinations.

Standard retrieval-augmented generation mechanisms typically function through a two-stage sequential architecture consisting of offline knowledge preprocessing and online context retrieval. In classic unimodal systems, source documents are split into discrete text segments, translated into dense vector representations via language models, and indexed within a vector database for similarity matching against encoded text queries. While effective for language-centric domains, this traditional structure breaks down when confronted with the dense visual notations of building design. Guidelines for fire protection, accessibility, and structural detailing are routinely communicated through diagrams, annotated sections, and matrix tables where graphic cues convey complete technical requirements independently of accompanying captions. When classical pipelines encounter these assets, they either discard graphical elements entirely or reduce them to simplistic textual summaries. This systemic omission of visual geometry impairs artificial intelligence assistance in engineering, leaving practitioners vulnerable to incorrect interpretations of regulatory constraints, flawed structural assumptions, and severed links between generated answers and authoritative reference documents.

To overcome these historical limitations without introducing the prohibitive computational overhead of training specialized table encoders from scratch, the researchers implemented a dual-mode table decomposition and data curation strategy. The study constructed a curated architectural corpus drawing strictly upon authoritative engineering references, including the Uniform Standard for Design of Civil Buildings (GB50352-2019), the Chinese Architectural Design Manual compiled by the Architectural Society of China, professional textbooks, and verified real-world engineering cases, while deliberately excluding informal internet blogs. During data preprocessing, each complex table is bifurcated into parallel representations: vision-language models, such as Qwen2.5-VL, semantically parse and extract each table row into an enriched natural language snippet that is concatenated into an augmented textual paragraph, while the original table is simultaneously rendered as a high-resolution visual screenshot. Data preparation incorporates automated visual filtering alongside targeted manual corrections, including the redrawing of degraded legacy drawings and the manual verification of merged cells, ensuring that geometric annotations, dimensional measurements, and category hierarchies remain completely intact.

The underlying retrieval engine establishes cross-modal grounding by mapping textual corpora and visual assets into a shared semantic embedding space. Raw textual passages are tokenized, normalized, lemmatized, and passed through a dedicated textual encoder to produce dense language embeddings, while visual inputs are dimensionally standardized, normalized, and processed through a dedicated visual encoder. The resulting modality-specific representations are unified through a weighted feature fusion function, expressing joint embeddings as the linear sum of scaled textual and visual vectors controlled by specific weighting coefficients. When a user submits an input query, the text is encoded into the shared semantic space, allowing the system to compute cosine similarity scores between the query vector and all stored multimodal representations in the database. By projecting disparate information types into an identical semantic coordinate space, the architecture allows pure text queries, image inputs, or mixed-modality prompts to locate corresponding visual diagrams and textual explanations through unified vector distance measurements.

Selecting an effective embedding architecture proved central to achieving reliable cross-modal alignment across technical building representations. The authors executed a comparative experimental evaluation examining three multimodal embedding models: OpenAI’s CLIP (using clip-vit-large-patch14 with 768-dimensional embeddings), VisualBERT (using visualbert-vqa-coco-pre with 768 dimensions), and ByteDance’s Doubao (using doubao-embedding-vision-250615 with 2048 dimensions). Across an evaluation set comprising 80 paired text-image examples and 20 text-only architectural samples, CLIP recorded an average text-image cosine similarity score of 0.25 with a standard deviation of 0.04, while VisualBERT achieved an improved similarity score of 0.46 with a standard deviation of 0.08. Benefiting from its higher-dimensional vector space, Doubao substantially outperformed both baselines, registering the highest average text-image similarity of 0.55 with a standard deviation of 0.07. Rigorous two-sample statistical testing confirmed these performance disparities were statistically significant at p < 0.001, with Doubao demonstrating marked superiority over CLIP (t = 30.69) and VisualBERT (t = 6.96), leading to its selection as the framework’s core embedding model.

To guarantee operational precision when handling unpredictable user phrasing, the framework incorporates an adaptive query preprocessing and domain-gating module. Natural language inputs undergo systematic normalization to strip away conversational fillers, polite phrasing, and interrogative preambles such as “Can you provide” or “What are,” distilling prompts down to essential semantic tokens. Controlled experimental testing demonstrated that progressively refining verbose prompts into normalized queries elevated cosine similarity against source technical text from 0.62 to 0.72, delivering an immediate alignment improvement of approximately 10%. To prevent the retrieval engine from returning irrelevant data when confronted with out-of-domain questions, the researchers established an operational similarity threshold denoted as theta at 0.5. Empirical evaluations of deliberately irrelevant queries containing overlapping vocabulary—such as prompts regarding crowded elevators, elderly care, or retail fire equipment—consistently exhibited similarity scores below 0.5. When top-ranked database matches fall below this boundary, the system suppresses irrelevant retrieval, issues an out-of-domain advisory, and deploys the top three nearest embeddings through a language model to generate clarifying guidance questions.

Once candidate knowledge pairs pass the relevance threshold, the highest-ranked multimodal entry is forwarded to an advanced multimodal large language model for contextual response generation. Rather than overwhelming the generative model with extensive context windows, the architecture isolates the top-ranked visual and textual pair, directing an interpretative multimodal model to conduct a query-conditioned visual inspection of the raw image. This visual interpretation stage extracts critical quantitative dimensions, structural configurations, and embedded mathematical expressions that are omitted from accompanying textual captions. Subsequently, the user query, verified source text, extracted visual insights, and raw image are assembled into a structured prompt processed by a reasoning-enabled multimodal model, such as Qwen3. This two-stage generative pipeline ensures that the final response is strictly grounded in verified database evidence, allowing the model to construct comprehensive technical explanations, highlight subtle engineering relationships, and dynamically formulate suggested follow-up queries for extended exploration.

The practical utility of the framework was validated across several demanding engineering scenarios where general-purpose commercial foundation models demonstrated distinct failure modes. In structural engineering benchmarks, the authors queried the system for the bending moment diagram of a cantilever beam under uniform loading. Comparative evaluations conducted in July 2025 revealed that frontier general-purpose models, specifically GPT-4.1 and Gemini 2.5 Pro, generated technically inaccurate diagrams displaying erroneous curvature and incorrect moment distribution profiles. In contrast, the M-RAG framework retrieved the verified engineering diagram from its curated database and paired it with a grounded textual breakdown explaining the governing mechanical principles. Similarly, when tested on regulatory fire separation distances sourced from building code GB50352-2019, the dual-mode indexing pipeline retrieved an explanatory technical synthesis, an exact reference table delineating building grade requirements, and an annotated spatial diagram, demonstrating how complex regulatory compliance checks can be rendered transparent and immediately verifiable.

To evaluate cross-modal versatility, the team conducted retrieval experiments using an accessibility dataset of washbasin installations for elderly housing derived from the Chinese Architectural Design Manual. The vector database maintained three separate representation categories: text-only embeddings, image-only embeddings, and paired text-image embeddings. Specialized text queries closely aligned with the dataset achieved a cosine similarity of 0.768 against text-only vectors, while specialized image queries reached 0.686 against image-only vectors, demonstrating strong modality-specific affinity. Combined multimodal queries maintained balanced, robust similarity values across all representations, scoring 0.231 against text, 0.218 against image, and 0.252 against paired embeddings. The researchers also mapped user interaction modalities into depth-first and breadth-first discovery pathways. In depth-first sequences, users progressively refined simple beam queries into advanced structural comparisons, identifying non-obvious scenarios where a cantilever beam of length L/2 exhibits an identical maximum bending moment of qL^2/8 as a simply supported beam of length L. In breadth-first sequences, layout investigations smoothly branched into passive solar sunspaces and acoustic properties of double-glazed glass.

Despite its proven technical retrieval and reasoning accuracy, the authors cataloged several operational limitations that outline important trajectories for future computer-aided design research. The current M-RAG architecture lacks native support for complex spatial engineering formats, such as object-based Building Information Modeling files and temporal construction video feeds, currently relying on static two-dimensional exports and screenshots that discard underlying geometric parameters, topological hierarchies, and dynamic construction stages. Future developments must focus on formulating geometry-aware encoders capable of directly parsing parametric BIM models, transitioning the software from a passive retrieval system into an active, autonomous artificial intelligence agent capable of managing multi-step project tasks, and establishing live data-exchange links with digital twins across the Architecture, Engineering, and Construction lifecycle. Such advanced integrations will allow multimodal retrieval systems to proactively execute clash detections, push parameter updates, and provide human-inspectable reasoning traces across every phase of building design and construction.

Subject of Research: Multimodal retrieval-augmented generation for architectural engineering knowledge

Article Title: Multimodal retrieval-augmented generation framework for visually rich knowledge in the architecture domain

Article References: Meng, X., & Tong, Z. (2025). Multimodal retrieval-augmented generation framework for visually rich knowledge in the architecture domain. Architectural Intelligence, 4(1), Article 22. https://doi.org/10.1007/s44223-025-00102-6

Image Credits: AI Generated

DOI: 10.1007/s44223-025-00102-6

Keywords: Multimodal RAG, Architectural Intelligence, Vision-Language Models, Semantic Embedding, Building Engineering, Table Decomposition, Technical Retrieval, Artificial Intelligence, Multimodal, retrieval-augmented, generation, framework