Memory-Guided AI Learns What Normal Looks Like to Spot Surveillance Anomalies

Surveillance cameras watch over train stations, campuses, hospitals, and city streets around the clock, yet the overwhelming majority of what they record is utterly unremarkable: people walking, cycling, parking, and pausing. Buried inside that ocean of routine footage are the rare moments that matter—a car swerving onto a sidewalk, a person sprinting the wrong way down a crowded corridor, an abandoned bag left behind. Training software to recognize such events is notoriously difficult precisely because anomalies are, by definition, scarce and unpredictable. Nobody can compile a catalogue of every possible abnormal behavior, and labeling the few examples that do exist is expensive and incomplete. A new study published in Complex & Intelligent Systems by Santosh Prakash Chouhan, Narinder Singh Punn, and Mahua Bhattacharya of the Atal Bihari Vajpayee Indian Institute of Information Technology and Management, Gwalior, tackles this problem from a different angle: instead of teaching a network what abnormal looks like, it teaches the network to model normality so thoroughly that anything unusual betrays itself through a failed prediction.

The approach, called RMTA-Net, belongs to a family of techniques known as prediction-based unsupervised video anomaly detection. The core idea is elegant. The network is shown a short clip of video frames and asked to predict what the next frame will look like. Because it is trained only on normal scenes, it becomes an expert at forecasting ordinary motion and appearance. When an anomalous event unfolds, the model’s prediction diverges sharply from the actual frame, and the size of that discrepancy—measured pixel by pixel—becomes the anomaly score. Frames that deviate strongly from expectation are flagged for human review. This sidesteps the need for exhaustive annotation, since the model learns the statistics of normality from unlabeled footage, but it places enormous demands on the network’s ability to represent both what objects look like and how they move through time.

Existing prediction-based methods, the authors argue, fall short on two fronts. First, their spatial representations—the internal descriptions of shapes, textures, and objects in each frame—are often too coarse to capture subtle appearance variations. Second, they struggle to understand clip-level temporal dependency: the way motion in one frame conditions and constrains motion in the next. Complex scenes contain overlapping movements, occlusions, and camera perspective shifts that make naive temporal modeling brittle. A pedestrian partially hidden behind a van, a crowd flowing in two directions at once, or a vehicle entering the frame at high speed all require the model to reason jointly across space and time rather than treating each frame in isolation. RMTA-Net was designed specifically to close both gaps within a single architecture.

The first stage of the pipeline is a residual spatial feature enhancement network. Operating on individual frames, it progressively extracts structural and appearance information at multiple levels of abstraction, from fine edges and textures to coarse object layouts. The residual design means that each processing stage refines the features produced by the previous one, adding detail rather than replacing it, which helps preserve the subtle cues that distinguish a person from a shadow or a stroller from a bicycle. These independently encoded spatial features are then organized into an explicit temporal representation, effectively stacking the per-frame descriptions into a sequence that downstream modules can interrogate for motion patterns and inter-frame relationships.

The heart of the system is the module that gives the network its name: recurrent conditioned memory-guided temporal attention, or RMTA. This component fuses two ideas that have usually been pursued separately. The first is recurrent temporal processing, in which a recurrent network maintains a running internal state as it steps through the frames of a clip, carrying forward information about what has already happened. The second is a set of learnable memory banks—trainable storage that encodes dataset-level normality priors, essentially compressed prototypes of how normal scenes tend to look and move. During processing, the model performs attention-based memory access, querying the memory banks with the recurrent network’s output to retrieve relevant normality prototypes, while a parallel branch uses temporal attention to retrieve a global normality prior driven by the contextual relationships among frames in the clip.

Crucially, the two branches do not simply compete; they are combined through adaptive gating, a learned mechanism that decides, at each moment, how much weight to give the recurrent memory retrieval versus the contextual temporal relationships. When motion is regular and predictable, the recurrent pathway may dominate; when the scene is complex or the recurrent state is uncertain, the memory-guided global prior can compensate. This dual-branch design allows the model to maintain a robust sense of normality even under the appearance variations and tangled motion dynamics that degrade earlier methods. The fused representation is then passed to an attention-enhanced decoder, which reconstructs the future frame from the learned spatiotemporal embeddings. The decoder’s attention mechanism helps it focus on the regions of the scene most relevant to the prediction, sharpening the contrast between well-forecast normal content and the surprising content that signals an anomaly.

The authors evaluated RMTA-Net on three widely used benchmark datasets that together span the practical challenges of surveillance video. UCSD Ped2 contains pedestrian walkways filmed from a fixed viewpoint, with anomalies such as cyclists and small vehicles intruding into pedestrian-only spaces. CUHK Avenue features a campus avenue with more varied activities, including loitering, throwing objects, and unusual walking patterns. ShanghaiTech is the most demanding of the three, with thirteen scenes of differing complexity, crowded conditions, and diverse anomaly types. On these benchmarks, RMTA-Net achieved frame-level area-under-the-curve scores of 99.0 percent on UCSD Ped2, 90.1 percent on CUHK Avenue, and 75.8 percent on ShanghaiTech, remaining competitive with several state-of-the-art methods across the board. The AUC metric reflects how well the system ranks anomalous frames above normal ones, so scores approaching 99 percent on Ped2 indicate near-perfect discrimination in that relatively controlled setting, while the ShanghaiTech result demonstrates useful performance in genuinely cluttered, multi-scene environments.

What makes this result notable is not merely the headline numbers but the architectural lesson embedded in them. Memory-based reasoning gives the network something like an institutional memory of normality: rather than relying solely on the hidden state of a recurrent network, which must compress an entire clip into a fixed-size vector, the model can consult explicit prototypes of normal appearance and motion. Attention-guided retrieval means those prototypes are consulted selectively, only where the current context calls for them. The adaptive gate then arbitrates between memory and context on the fly. This division of labor mirrors, in a loose computational sense, the way long-term and episodic memory support expectations in human perception, a parallel the authors’ choice of terminology makes explicit. The result is a system whose predictions degrade gracefully under complexity rather than collapsing when scenes become crowded or visually ambiguous.

The practical implications extend well beyond benchmark scores. Because the method requires no anomaly labels, it can be deployed on the vast reservoirs of unlabeled footage that organizations already possess, learning the normal rhythm of a specific environment and flagging deviations without prior knowledge of what those deviations might be. That matters for applications ranging from public safety and traffic monitoring to industrial safety and elder care, where the set of possible incidents cannot be enumerated in advance. The open-access publication, with fees covered by the Government of India’s One Nation One Subscription initiative, means the full technical description is freely available to researchers and practitioners worldwide, lowering the barrier for replication and extension.

Challenges remain, as the authors’ own results make clear. Performance on ShanghaiTech, while competitive, still leaves substantial room for improvement in the hardest scenes, and future-frame prediction systems in general can be sensitive to camera jitter, illumination shifts, and the sheer diversity of real-world behavior. The field will also need to keep examining how such models behave across different populations and environments to ensure that flagged anomalies reflect genuine events rather than artifacts of the training distribution. Nevertheless, RMTA-Net offers a compelling demonstration that combining recurrent temporal modeling, learnable memory, and adaptive attention yields a more faithful model of normality—and that the better a machine understands the ordinary, the more sharply the extraordinary stands out. In surveillance, where the abnormal is rare by definition, that principle may prove to be the key to systems that finally earn the trust placed in the cameras they watch.

Subject of Research: Unsupervised video anomaly detection using future-frame prediction with memory-guided temporal attention

Article Title: RMTA-Net: recurrent conditioned memory-guided temporal attention with spatial feature enhancement for video anomaly detection

Article References: Chouhan, S. P., Punn, N. S., & Bhattacharya, M. (2026). RMTA-Net: recurrent conditioned memory-guided temporal attention with spatial feature enhancement for video anomaly detection. Complex & Intelligent Systems. https://doi.org/10.1007/s40747-026-02506-x

Image Credits: AI Generated

DOI: 10.1007/s40747-026-02506-x

Keywords: video anomaly detection, future-frame prediction, recurrent neural networks, memory banks, temporal attention, spatial feature enhancement, unsupervised learning, surveillance video, computer vision, deep learning, UCSD Ped2, ShanghaiTech