Robots That Learn From Old Data: Review Maps Offline Reinforcement Learning Under Data Scarcity

Teaching a robot arm to grasp, insert or stack objects through live trial and error is expensive, slow and often unsafe, so many researchers train policies offline, from datasets recorded long before the robot ever moves. A new narrative review, published open access in Discover Data on 15 October 2026, examines how offline reinforcement learning behaves when those datasets are small, narrow and noisy rather than large and diverse. The authors, a team led by Ghusoon Hadi AL-aldaffaie with colleagues including Deniz Noormohammadzadeh Maleki, deliberately frame the work as a focused narrative review rather than a systematic one, stating plainly that it does not follow a PRISMA protocol and does not claim exhaustive coverage of the literature.

The review’s central organizing idea is an operational definition of what the authors call data-limited robotic manipulation. A setting qualifies if it meets at least one of five conditions: too few transition tuples to cover the relevant state-action space, low behavioral diversity with fewer than three distinct manipulation strategies, reward signals informative in fewer than ten percent of transitions, coverage restricted to a small subregion of the workspace, or demonstrations collected under systematic constraints such as limited operator skill, scripted motion or safety monitors that cut episodes short. As reference points, the authors cite the D4RL pen-cloned variant, with roughly 5,000 transitions and a success rate below 40 percent, as a borderline case, and the relocate-cloned variant, with a success rate below 10 percent, as severely data-limited.

Why does scarcity matter so much? Offline reinforcement learning inherits the classic failure modes of learning from fixed data, and the review argues these intensify when data is thin. Extrapolation error arises when a learned policy selects actions poorly represented in the dataset, forcing the Q-function to estimate their value without supporting experience, which can produce overly optimistic estimates and unstable robotic behavior. Bootstrapping error accumulation compounds the problem, because temporal-difference updates repeatedly propagate estimation errors without any corrective interaction from the environment. In sparse-reward tasks such as grasping, where reward arrives only on successful completion, credit assignment across the manipulation sequence becomes especially fragile.

The review then sorts the field into six algorithmic families plus two enabling directions. Conservative value-based methods such as CQL add a pessimism penalty that minimizes Q-values on out-of-distribution actions while maximizing them on observed actions, effectively learning a lower bound on the true value function in unobserved regions. IQL takes a different route, avoiding out-of-distribution queries entirely by replacing Bellman maximization with in-sample expectile regression. The authors note that these methods assume unseen actions are on average suboptimal or unsafe, an assumption reasonable in safety-critical manipulation but potentially restrictive when the dataset contains only suboptimal demonstrations.

Each family carries its own low-data failure mode, and the review is unusually explicit about them. Under severe scarcity, CQL’s conservatism penalty can become globally dominant, collapsing Q-values across the whole action space and producing a near-zero policy that avoids everything, a phenomenon the authors call over-conservatism or policy collapse, particularly acute when successful trajectories make up fewer than five percent of the dataset. IQL avoids explicit penalties but may fail to improve beyond the median demonstrated return when no high-quality demonstrations exist. Behavior-constrained methods such as BCQ, BRAC and TD3+BC can become trapped in the demonstrated behavior manifold; in TD3+BC, insufficient diversity lets the behavior-cloning term dominate, reducing the method to supervised imitation of potentially suboptimal demonstrations.

Advantage-weighted approaches such as AWAC and AWR sit between imitation and policy improvement, weighting actions by estimated advantage. The review warns that in sparse-reward grasping, where most transitions carry zero reward, the exponential weighting scheme amplifies small estimation errors, causing the policy to overfit to a handful of accidentally high-reward transitions. The authors are careful to flag that no published threshold establishes exactly where this failure becomes severe in manipulation datasets, so any specific number should be read as a heuristic rather than an established finding. Uncertainty-aware methods, including RFQI and ensemble approaches, model epistemic uncertainty to penalize updates in poorly covered regions, though the authors caution that ensemble disagreement may become a less reliable uncertainty proxy when all members train on the same small dataset, a plausible extrapolation from broader deep-learning findings rather than a demonstrated result.

Model-based methods such as MOPO and MOReL attack scarcity directly by learning an approximate dynamics model and generating synthetic rollouts to expand the effective training set. MOPO penalizes rollouts entering high-uncertainty regions of the learned model, while MOReL restricts planning to regions where model confidence exceeds a threshold, a conservative strategy aligned by design with hardware safety. The review expects learned dynamics models to degrade sharply near contact events, precisely where grasp success is decided, because contact forces are discontinuous and highly sensitive to end-effector pose; the authors label this a mechanistic extrapolation from the general model-based RL literature, not a manipulation-specific empirical result, and call for direct validation on contact-rich benchmarks. Model exploitation, where a policy achieves high simulated returns by exploiting systematic model errors that do not exist in the real environment, remains a key risk.

Sequence-modeling approaches, represented by the Decision Transformer, reframe offline reinforcement learning as conditional sequence prediction, bypassing the Bellman update entirely. The review argues that such methods are theoretically unable to perform trajectory stitching, combining fragments of suboptimal trajectories into a policy better than any single demonstration, a limitation established by Brandfonbrener and colleagues and confirmed empirically by others. Because data-limited manipulation datasets by definition contain few high-return trajectories, the authors treat this as a reasoned extrapolation to manipulation, not a manipulation-specific finding, and note it has not been tested on contact-rich benchmarks. They suggest sequence models suit long-horizon tasks such as multi-step assembly, while value-based methods may be more data-efficient for reactive, near-Markovian grasping, again presented as analytical judgment rather than demonstrated comparison.

The quantitative backbone comes from D4RL and NeoRL benchmarks. On the D4RL hopper-medium-replay dataset, widely regarded as a proxy for limited-coverage, mixed-quality data, CQL achieves normalized returns of approximately 86 to 95, whereas BCQ achieves 55 to 75 under comparable settings. IQL exceeds 85 on antmaze-medium-play, outperforming TD3+BC at roughly 48. On NeoRL’s low-data split, CQL degrades fewer than 10 normalized-return points from its full-data baseline while AWAC degrades more than 20 points. MOPO and MOReL surpass behavior-constrained baselines by 5 to 15 points on D4RL locomotion tasks with limited interaction budgets, but this advantage narrows or reverses on contact-rich variants such as pen-cloned and relocate-cloned. The Decision Transformer reaches roughly 95 on hopper-expert but drops below 65 on mixed-quality data, revealing high sensitivity to behavioral diversity.

The review is candid about its limits. Because different studies use varying datasets, environments, protocols and metrics, direct numerical comparison is difficult, and the authors present their conclusions as benchmark-informed comparative trends rather than precise rankings. Benchmark environments themselves simplify real robotics, omitting sensor noise, partial observability, calibration errors, latency and unstable contact dynamics. Looking forward, the authors identify tactile-aware uncertainty estimation, physics-informed world models embedding rigid-body contact constraints, foundation models augmented with physical priors, control barrier functions as formal safety filters for offline-to-online transitions, and purpose-built low-data manipulation benchmarks as the directions most likely to close the gap between benchmark success and reliable physical deployment. No single method, they conclude, fully addresses the compounded challenges of learning to move matter safely from very little data.

Subject of Research: Offline reinforcement learning for robotic manipulator control under data-limited conditions

Article Title: A review of offline training of robotic manipulator control under limited data

Article References: AL-aldaffaie, G. H., Ali, Z. M., Farhadi, A., Zamanifar, A., & NoormohammadzadehMaleki, D. (2026). A review of offline training of robotic manipulator control under limited data. Discover Data, 4(1), Article 22. https://doi.org/10.1007/s44248-026-00120-3

Image Credits: AI Generated

DOI: 10.1007/s44248-026-00120-3

Keywords: offline reinforcement learning, robotic manipulation, CQL, Decision Transformer, MOPO, MOReL, D4RL, NeoRL, data scarcity, model-based RL, offline-to-online learning, robot grasping