Reinforcement learning for long-horizon robot manipulation mapped in 130-study review

Robots that can stack blocks, assemble parts, or tidy a kitchen over dozens of sequential actions are still surprisingly hard to train, and a new large-scale survey of the field shows why: researchers have been solving the same narrow slice of the problem over and over while leaving its hardest corners nearly untouched. A scoping review published on 28 July 2026 in the journal Discover Robotics analyzed 130 studies on reinforcement learning for long-horizon robotic manipulation, all published between 2020 and 2026, and mapped exactly where the field’s effort has piled up and where it has stalled. The review was conducted by Matthew Acs and Xiangnan Zhong and followed the PRISMA-ScR reporting guidelines for scoping reviews.

Long-horizon manipulation is the branch of robotics in which a machine must complete tasks that unfold across many dependent steps: multi-object rearrangement, assembly, tool use, or household activities in which subgoals must be reached in a specific order. The authors define the problem at the level of the task itself, not the algorithm. A task counts as long-horizon when success requires temporally extended decision-making across multiple dependent actions, subtasks, skills, or contact events, with delayed and often sparse feedback. That structure creates well-known difficulties for reinforcement learning, the technique in which agents learn control policies from interaction rewards rather than explicit programming. Credit for a final success must be assigned backward through a long chain of earlier decisions, exploration becomes expensive when meaningful reward arrives only at completion, and the agent must integrate perception, planning, and control across very different time scales.

To build the evidence map, the team searched five databases, IEEE Xplore, the ACM Digital Library, Scopus, Web of Science, and PubMed, in an original search covering January 2020 through October 2025, then ran an updated search through June 2026. The original search returned 1,316 records; after deduplication, title and abstract screening, and full-text assessment against predefined eligibility criteria, 104 studies remained. The update added 26 more, for a total of 130. Studies were excluded if they focused on navigation, locomotion, aerial robots, or multi-agent coordination, or if their primary contribution centered on large language models, foundation models, or vision-language-action systems rather than reinforcement learning itself. Single-step grasping and short reactive control tasks were also ruled out. The authors stress that the review was not prospectively registered and that formal independent duplicate screening and inter-reviewer agreement statistics were not used, so some coding decisions involved individual interpretation.

The headline numbers describe a field growing fast but narrowing in focus. Annual publications rose from 5 studies in 2020 to 41 in 2025, more than an eightfold increase, with growth accelerating between 2020 and 2023 before stabilizing in 2024. Work concentrates in robotics venues: the International Conference on Robotics and Automation leads with 22 papers, followed by IEEE Robotics and Automation Letters with 16, the Conference on Robot Learning with 10, and IROS with 9. Journal output, rare early on, approached parity with conference papers by 2024, which the authors read as a sign of maturation.

The clearest pattern is architectural. The review organizes the literature along three structural axes, temporal abstraction, predictive modeling, and structural guidance, and identifies four recurring archetypes. Hierarchical methods dominate, appearing in 45.4 percent of studies (59 of 130). Most of these use a two-level design, seen in 33.1 percent of the corpus, in which a high-level policy picks subgoals or skills and a low-level controller executes them; hierarchies of three or more levels appeared in exactly one study. Hybrid or structured architectures, which embed reinforcement learning inside planners, task graphs, or analytical controllers, account for 26.2 percent. Purely model-free or offline end-to-end methods make up 22.3 percent, and model-based approaches that learn explicit dynamics models for planning are the rarest at 6.2 percent. Notably, most systems combine established actor-critic algorithms such as SAC and PPO with planning modules or skill abstractions rather than introducing fundamentally new learning algorithms, suggesting progress has come from architectural integration.

The task and sensing landscape is even more skewed. Block and object stacking or rearrangement tasks form the largest category with 42 studies, followed by general manipulation with 36, assembly and insertion with 23, kitchen and household multi-step tasks with 12, and dexterous in-hand manipulation with 9. Deformable object manipulation and mobile manipulation appear in only 4 studies each. On the perception side, state-based observations dominate with 73 studies, compared with 20 using vision alone and 19 combining state and visual input; tactile and multimodal configurations are rare. End-effector control is the most common action representation at 64 studies, and the single most frequent configuration in the entire corpus is block rearrangement with state observations and end-effector control, a template well supported by simulators that hand the agent exact object poses.

Evaluation practices mirror that concentration. Of the 130 studies, 72, or 55.4 percent, incorporated sim-to-real transfer, 53, or 40.8 percent, were simulation-only, and just 5, or 3.8 percent, relied exclusively on real-world learning without simulation. Experiments cluster around a handful of simulators, including PyBullet, MuJoCo, robosuite, NVIDIA Isaac, and Franka Kitchen, and around two robot families, the Franka Panda and UR-series arms. Binary task success is by far the most reported metric, but secondary measures such as subtask success, episode length, force and contact measures, and robustness vary widely with task type. The authors also flag deep inconsistency in how “long horizon” itself is defined, with some papers counting environment steps, others counting subtasks, search depth, or even hours of autonomous operation, making cross-study comparison of task difficulty nearly impossible.

From these patterns the review builds a “gap atlas” of underexplored directions. Real-world-only learning, reset-free execution in which robots must recover from failures without manual resets, dexterous and deformable manipulation, mobile manipulation, multimodal perception, and model-based and offline reinforcement learning all rank among the least represented combinations. The authors caution that some of these gaps may reflect genuine difficulty, including complex contact dynamics, hardware cost, and safety constraints, rather than simple neglect, and that computational costs were not consistently reported across the surveyed studies. They also note that because success rates and sample-efficiency figures are defined over heterogeneous tasks, simulators, and reward structures, the review treats them as study-specific outcomes and performs no statistical meta-analysis or effect-size pooling.

As a remedy, the authors propose standardized reporting recommendations covering six dimensions: explicit task structure and horizon definitions, standardized evaluation protocols and metrics beyond aggregate success rates, stated reset assumptions and execution constraints, task domain and diversity, perception and action representations, and detailed hardware and deployment reporting for physical robots. The review’s limitations are stated plainly: it covers only peer-reviewed publications in English, excludes recent preprints, reflects the literature only through the June 2026 search date, and relies on coded descriptions rather than independently re-run experiments. Even so, the evidence map delivers a clear message: long-horizon robotic learning has matured into a sustained research area built largely on shallow two-level hierarchies, simulated block rearrangement, and state-based control, and its next leap toward genuinely deployable robots will likely require broader tasks, richer sensing, and learning that survives the messiness of the real world.

Subject of Research: A scoping review mapping reinforcement learning methods, task domains, and evaluation practices for long-horizon robotic manipulation across 130 studies published from 2020 to 2026.

Article Title: A scoping review of reinforcement learning methods for long horizon robotic manipulation

Article References: Acs, M., & Zhong, X. (2026). A scoping review of reinforcement learning methods for long horizon robotic manipulation. Discover Robotics, 2(1), Article 22. https://doi.org/10.1007/s44430-026-00037-1

Image Credits: AI Generated

DOI: 10.1007/s44430-026-00037-1

Keywords: reinforcement learning, robotic manipulation, long-horizon tasks, hierarchical reinforcement learning, scoping review, sim-to-real transfer, model-based RL, offline RL, reset-free learning, dexterous manipulation, robot benchmarks, PRISMA-ScR