A new anomaly-detection system could give reinforcement-learning agents an early warning when the world around them begins to change, without inspecting the agents’ policies or relying on obvious visual glitches. The method, called Imagined Trajectory Representation Matching, or ITRM, uses a learned “world model” to predict how an environment should evolve and then checks whether the unfolding situation remains close to patterns associated with normal operation. In tests spanning navigation, robotic control, manipulation and autonomous driving, the approach achieved an average area under the receiver operating characteristic curve (AUROC) of 0.853 and a false-positive rate of 0.279 at 95 percent true-positive sensitivity. The results suggest that monitoring an agent’s internal policy may not be necessary to detect dangerous changes in its surroundings.
Reinforcement learning systems learn by interacting with an environment: they observe a situation, choose an action and receive feedback before repeating the cycle. This framework has enabled impressive progress in robotics, games and autonomous control, but it also creates a safety problem. A system trained under one set of conditions may encounter a failed sensor, a degraded actuator, a changed surface, an unexpected obstacle or a disturbance in the environment after deployment. Such changes may not look strange in a single image. Instead, they may reveal themselves through altered cause-and-effect relationships—for example, a vehicle that no longer turns as expected or a robot whose movement produces a different response from the one predicted during training. Detecting these shifts requires more than identifying unusual pixels; it requires understanding temporal dynamics.
Traditional anomaly detectors often model the distribution of individual observations. In image-based reinforcement learning, however, an image alone does not reveal whether a difference comes from the environment, the agent’s behavior or ordinary visual noise. The researchers therefore trained a recurrent state-space model, or RSSM, using only episodes collected under normal conditions. The model compresses high-dimensional images and action sequences into a lower-dimensional latent space, where abstract variables represent the hidden state of the environment. Its recurrent component maintains a deterministic memory of recent events, while a stochastic latent component represents uncertainty about what is happening. By learning how these variables evolve over time, the model builds a predictive description of normal environment behavior.
The RSSM distinguishes between what it expects to happen before seeing the next observation and what it infers after seeing that observation. Its predictive prior estimates the next latent state from the recurrent history and previous actions, while its posterior updates that estimate using the newly arrived image. During training, the two distributions are encouraged to agree through a Kullback–Leibler divergence term, while an observation-decoder component learns to reconstruct images from the latent state. The system is not being trained to recognize anomalies directly. Instead, it learns the structure of nominal episodes and later uses that structure as a reference. This separation is important because the safety monitor can be trained after the reinforcement-learning policy is already fixed.
At deployment, ITRM extracts deterministic embeddings from the world model rather than comparing raw images. These embeddings can include the recurrent hidden state, which summarizes the environment’s predictable temporal structure, and statistics of the predictive prior, such as its mean or mode. The method then stores embeddings produced by normal training trajectories in a feature bank. For each new embedding, it calculates the average distance to its nearest neighbors in that bank. A normal state should lie near many previously observed normal states; a shifted or anomalous state should be more isolated. The resulting nearest-neighbor distance becomes an anomaly score. If the score rises above a threshold calibrated with nominal validation data, the system flags a possible environmental change.
This strategy avoids several weaknesses of alternative approaches. Detectors based on a policy network’s uncertainty or hidden features are coupled to a particular agent and may fail when the policy or reinforcement-learning algorithm changes. Reconstruction-based systems can be distracted by perceptual details that have little bearing on control, such as lighting or texture. Methods designed for ordinary time-series data generally cannot handle the dimensionality of images. ITRM instead uses the world model as an external dynamics monitor: it does not need access to policy internals, anomaly-specific labels or examples of every possible failure. Because it compares compact latent representations, it is intended to identify deviations in the underlying Markov decision process, including changes to observations, actions or transition dynamics.
The researchers evaluated ITRM on Anomaly-Gym, a benchmark containing 10 continuous-control environments and 12 anomaly types affecting observations, dynamics or actions. The environments include navigation tasks, robotic control, robotic manipulation and autonomous driving. Models were trained on 100 normal episodes per task and tested on episodes in which anomalies appeared at randomized times and with either tiny or strong severity. Compared with computer-vision baselines based on autoencoder reconstruction, pretrained ResNet features, predictive coding, DINO features and CLIP embeddings, as well as competing reinforcement-learning and world-model detectors, ITRM produced the strongest aggregate AUROC and lowest average FPR95. The gains were broad rather than being confined to one environment, although performance varied substantially between tasks.
The system also tended to detect anomalies sooner after they appeared, particularly within the first five, 10 and 20 time steps, but the experiments exposed a major obstacle to real-world use: threshold selection. Fixed thresholds based on three standard deviations or the 95th percentile of normal scores sometimes missed anomalies, while adaptive exponentially weighted thresholds reduced missed detections at the cost of more alerts before an anomaly actually occurred. The detector transferred across policies, achieving AUROC values of 0.73 and 0.71 in tests involving policies not used for training, but its false-positive rate nearly doubled because the new policies visited different regions of the environment even under normal conditions. The authors also found that deterministic features and expectation-based latent rollouts were more reliable than sampled alternatives, that one nearest neighbor performed best in their experiments, and that a compressed k-center feature bank retained an AUROC of 0.803 using only 5 percent of the full reference data. The findings point toward a practical safety layer for visual reinforcement-learning systems, while emphasizing that broader state coverage, robust calibration and testing under realistic visual noise will be essential before such monitors can be trusted in high-stakes deployments.
Subject of Research: Anomaly detection for image-based reinforcement-learning agents using latent representations from recurrent world models.
Subject of Research: Technology and Engineering
Article Title: Imagining Trajectories for Anomaly Detection in Reinforcement Learning from Images
Article References: Haider, T., Roscher, K., & Günnemann, S. (2026). Imagining Trajectories for Anomaly Detection in Reinforcement Learning from Images. Machine Learning, 115(9), Article 205. https://doi.org/10.1007/s10994-026-07126-7
Image Credits: AI Generated
DOI: 10.1007/s10994-026-07126-7
Keywords: anomaly detection, reinforcement learning, world models, image-based control, out-of-distribution detection, recurrent state-space models, robotics, autonomous systems
Cite Scienmag News
APA
MLA
Chicago
Florence R. (August 29, 2026). Imagining Trajectories Helps Detect Anomalies in Image-Based Reinforcement Learning. Scienmag. https://scienmag.com/imagining-trajectories-helps-detect-anomalies-in-image-based-reinforcement-learning/
Florence R. “Imagining Trajectories Helps Detect Anomalies in Image-Based Reinforcement Learning.” Scienmag, 29 August 2026, https://scienmag.com/imagining-trajectories-helps-detect-anomalies-in-image-based-reinforcement-learning/. Accessed 29 August 2026.
Florence R. “Imagining Trajectories Helps Detect Anomalies in Image-Based Reinforcement Learning.” Scienmag. August 29, 2026. https://scienmag.com/imagining-trajectories-helps-detect-anomalies-in-image-based-reinforcement-learning/
Copy citation
Download RIS
Tags: Anomaly detection in reinforcement learningautonomous agent monitoringautonomous driving environment monitoringautonomous navigation anomaly detectionearly warning system for RL agentsearly warning systems for robotic controlenvironment change detectionenvironment change detection in autonomous systemsenvironment evolution monitoring in autonomous drivingenvironment evolution predictionImagined Trajectory Representation MatchingReinforcement learning anomaly detectionreinforcement learning safetyrobotic control safetyrobotic manipulation safety monitoringrobust reinforcement learning under environmental changessafety in reinforcement learningtrajectory prediction for environment monitoringtrajectory-based anomaly detectionvisual anomaly detection in reinforcement learningvisual anomaly detection in roboticsworld model for environment predictionworld model-based anomaly detection
