Structured Memory Scheduling Helps Large Language Models Reason Beyond Retrieval

Retrieval-augmented generation has become the standard way to ground large language models in external knowledge, but a new open-access study argues that fetching relevant documents is only half the battle. Researchers publishing in Data Science and Engineering on 17 June 2026 propose SMS-RAG, a structured memory scheduling retrieval-augmented generation framework that treats external knowledge not as a passive pile of context but as an actively coordinated reasoning resource. Across three multi-hop question-answering benchmarks and a legal QA dataset, the authors report that SMS-RAG significantly outperforms state-of-the-art retrieval methods in reasoning accuracy, factual coverage, and scheduling efficiency.

The core problem the team identifies is not a lack of knowledge but its organization. Most existing RAG systems, whether they rely on vector similarity, knowledge graphs, or retrieved historical cases, follow the same retrieval-as-context paradigm: relevant fragments are fetched by semantic similarity, concatenated into a flat block of text, and handed to the model for generation. The authors illustrate the failure mode with a legal consultation example asking whether a company may dismiss an employee who never signed a labor contract during probation. Answering correctly requires first establishing that an employment relationship exists, then assessing the validity of the probationary period, and only then judging the legality of the dismissal and any compensation liability. Vector retrieval can surface the applicable provisions, graph retrieval can extract relational paths such as no contract leading to an established labor relationship leading to employer fault, and case retrieval can find precedents supporting compensation. Yet once these pieces are flattened into one context, premise-conclusion and rule-exception relationships vanish. The rule that employees can be dismissed during probation and the rule that without a written contract there is effectively no probation period appear as two equally valid facts, even though the second invalidates the first. A model leaning on surface relevance may prioritize the wrong rule and wrongly conclude the dismissal was lawful.

The authors argue that retrieval and reasoning serve fundamentally different objectives. Retrieval maximizes semantic relevance; reasoning demands a structured process that follows sequential dependencies and explicit logical priorities. When the two are coupled only through flat context concatenation, the structural constraints reasoning needs become obscured, and the language model is forced to reconstruct logical structure from flattened input, which the paper describes as often unreliable in complex scenarios. The framework draws inspiration from cognitive science, particularly Global Workspace Theory, which holds that intelligent behavior emerges from the dynamic interaction of multiple specialized information sources, with only task-relevant information selectively activated and propagated while irrelevant information is suppressed.

SMS-RAG models external knowledge as three types of compound memory modules combining neural and symbolic components. Semantic memory, a neural module, provides semantic abstraction through continuous vector representations. Structural memory, a symbolic module, models entity relationships and logical constraints by constructing a knowledge hypergraph, an extension of ordinary graphs that can capture higher-order, multi-entity relations rather than only binary links. Episodic memory, an experiential module, stores historical reasoning trajectories and decision processes, offering procedural priors for current tasks. According to the paper, naively concatenating these heterogeneous sources causes modality interference, where critical structural constraints are drowned out by irrelevant semantic noise.

Three components orchestrate how this knowledge is used. MemPlanner analyzes the user query, decomposes it into subtasks, and decides which memory modules each subtask needs, generating a task-aware MemoryPlan. It scores each subtask-module pair as a weighted trade-off between relevance and access cost, where relevance is estimated by prompting a language model as a multi-class classification problem and cost reflects retrieval latency and token consumption, with semantic memory cheapest, episodic moderate, and structural most expensive. A threshold filters out inefficient retrieval operations before they run. MemScheduler then converts the linear plan into a directed dependency graph in which retrieval nodes feed task-level aggregation nodes, and aggregation nodes of predecessor subtasks connect to those of dependent subtasks, so later subtasks begin only after earlier knowledge is fully integrated. Nodes execute in topological order, and the authors note the graph construction and sorting complexity is negligible compared with token-level computation in a language model.

MemRouter executes the plan, routing retrieval requests only to channels activated in the MemoryPlan. Semantic retrieval performs approximate nearest neighbor search in vector space. Structural retrieval traverses the knowledge hypergraph to locate high-order relations incident to key entities, tracing the topological structure needed for multi-hop reasoning. Episodic retrieval scores historical cases by both semantic similarity and the isomorphism of their reasoning logic, computed via graph edit distance, so retrieved cases are not merely relevant but procedurally viable. Retrieved results are fused through three operators: appending new facts as independent branches, refining ambiguous entries, and weighted voting to resolve conflicts between sources.

The experimental evaluation spans HotpotQA, MuSiQue, and 2WikimQA, all multi-hop benchmarks requiring integration across multiple Wikipedia sources, plus a Legal dataset built from UltraDomain covering provisions, precedents, and regulatory interpretations. Nine baselines were compared, including flat retrieval methods, structure-aware systems such as GraphRAG, HippoRAG, and HyperGraphRAG, the case-based ZEBRA, multi-source systems Hydra and ToG 2.0, and the planning-based Self-RAG. Metrics included exact match, F1, and an LLM-based judge scoring reasoning chains on dimensions such as comprehensiveness, logical coherence, and factuality. Experiments ran on a single NVIDIA RTX 4090 with GPT-4o-mini, Llama-3.1 70B, and Qwen3-8B as backbones. Against Vanilla RAG, SMS-RAG improved F1 on HotpotQA by 28.01 points with GPT-4o-mini, 24.35 with Llama-3.1 70B, and 19.99 with Qwen3-8B, and the authors report consistent gains across all models and tasks.

Ablation studies support each design choice. Removing any single memory type degrades performance, with excluding structural memory causing the largest drop on MuSiQue at minus 10.96 F1, and removing episodic memory costing 3.20 F1 points on Legal. Single-source variants score lower than multi-source fusion, with the best structural-only variant nearly ten F1 points below the full model on MuSiQue. Comparisons of scheduling strategies show that a fixed activation order, greedy top-k selection, and uniform concatenation of all sources all fall short, with greedy selection producing the steepest F1 decline on MuSiQue at minus 4.35 and uniform aggregation losing at least two points through information dilution. Hypergraph structuring also outperformed standard triple-based knowledge graphs and sentence-adjacency path graphs on both tested datasets. A cost analysis on HotpotQA found SMS-RAG controlled token usage and latency while maintaining high accuracy, which the authors attribute to structured scheduling keeping context size in check.

A case study from MuSiQue shows the mechanism in action. Asked when an explorer reached the city housing the headquarters of the only group larger than the record label of Vilaiyaadu Mankatha, the baseline ToG 2.0 correctly traced the label to Sony Music India and its parent Universal Music Group, but then misplaced UMG’s headquarters in New York and locked that error in as a premise, ultimately answering Giovanni da Verrazano arriving in 1524, a structurally coherent but factually wrong chain. SMS-RAG generated a staged reasoning plan, activated different memory types at each stage, and restructured retrieved knowledge through explicit integration strategies, correctly identifying Santa Monica and answering that Gaspar de Portolà arrived there on August 3, 1769. The authors conclude that optimizing retrieval relevance is a necessary but insufficient condition for complex reasoning, and that explicit organization and scheduling of knowledge during reasoning are essential.

Subject of Research: Structured memory scheduling for retrieval-augmented large language model reasoning

Article Title: Why Retrieval is not Enough: Structured Memory Scheduling for Large Language Model Reasoning

Article References: Yao, X., Zhao, X., Zhou, D., Bi, X., Sun, Y., & He, B.-G. (2026). Why Retrieval is not Enough: Structured Memory Scheduling for Large Language Model Reasoning. Data Science and Engineering. https://doi.org/10.1007/s41019-026-00361-x

Image Credits: AI Generated

DOI: 10.1007/s41019-026-00361-x

Keywords: retrieval-augmented generation, large language models, multi-hop reasoning, knowledge hypergraph, memory scheduling, SMS-RAG, semantic memory, structural memory, episodic memory, question answering, natural language processing, Retrieval