Hybrid CNN-Transformer AI Spots Anomalies in Surveillance Video With Record Accuracy

A world with roughly one billion surveillance cameras is no longer a distant prediction, and the sheer volume of footage those cameras generate has long outstripped the capacity of human operators to watch it all. Against that backdrop, a team of researchers in India has unveiled a new artificial intelligence framework that learns what normal behavior looks like in a video scene and flags anything that deviates from it, without ever being shown a single labeled example of an anomaly. The work, published in Cluster Computing, combines two of the most influential architectures in modern computer vision, convolutional neural networks and vision transformers, into a single autoencoder that reconstructs the everyday rhythm of a scene and stumbles visibly when something unusual happens.

The research, led by Vandana Pathak of Graphic Era Deemed to be University in Dehradun, together with Manoj Diwakar, Neeraj Kumar Pandey, Sanjay Roka and Prabhishek Singh, addresses a stubborn problem in video surveillance: anomalies are rare, varied and almost impossible to enumerate in advance. A cyclist cutting through a pedestrian zone, a person sprinting the wrong way down a crowded corridor, or an abandoned bag left on a plaza all look nothing alike, which makes supervised classification impractical. The dominant alternative, unsupervised anomaly detection, flips the task on its head. Instead of teaching a model to recognize trouble, it teaches the model to recognize normality so thoroughly that trouble becomes conspicuous by its absence.

The centerpiece of the new framework is a memory-augmented CNN-ConvViT autoencoder. Autoencoders compress an input into a compact latent representation and then attempt to reconstruct it, and when trained exclusively on normal footage they reconstruct familiar patterns well and unfamiliar ones poorly. The reconstruction error then serves as an anomaly score. The difficulty, well documented in prior work, is that a sufficiently powerful autoencoder learns to generalize too much, reconstructing even the anomalies it was never trained on, which erases the very signal the system depends on. Memory modules were introduced to counteract this by storing prototypical features of normal behavior and forcing the encoder to express every input as a combination of those stored prototypes, limiting the network’s ability to improvise.

What distinguishes the new approach is the design of its memory component, called the Temporal-Aware Prototype Memory Module, or TAPMM. Rather than treating memory as a static dictionary of spatial appearances, TAPMM explicitly learns prototypes of normal spatio-temporal behavior, capturing how scenes evolve over time rather than merely how they look in a single frame. This temporal awareness matters because many surveillance anomalies are defined by motion rather than appearance: a person walking calmly through a parking lot is unremarkable, while the same person running through it may warrant attention. By encoding temporal structure into the memory itself, the framework narrows the gap between what the model stores and what constitutes an anomaly in practice.

The architecture also tackles a complementary weakness. Convolutional neural networks excel at extracting local spatial detail, such as edges, textures and the shapes of individual objects, but their receptive fields limit their grasp of long-range relationships across a scene. Vision transformers, by contrast, use self-attention to model global context, letting every part of an image attend to every other part, but they can be less efficient at capturing fine-grained local structure. The proposed framework fuses convolutional blocks with Conv-ViT blocks, a hybrid in which convolutional operations and transformer attention are integrated so that local spatial details and global contextual dependencies are modeled jointly. This kind of hybridization reflects a broader trend in computer vision, following influential studies asking whether vision transformers see like convolutional networks and demonstrating that the two paradigms have complementary strengths.

Motion information enters the pipeline through a second clever design choice. Alongside the raw video frames, the researchers feed the network an additional input channel computed with Farneback optical flow, a dense optical flow algorithm that estimates the motion vector of every pixel between consecutive frames. Optical flow gives the model an explicit, pixel-level description of how everything in the scene is moving, independent of how it looks. By combining appearance information from the raw frames with motion information from the flow channel, the network can learn both appearance-based variations, such as an unexpected object, and motion-based variations, such as movement in the wrong direction or at an abnormal speed. This dual-stream strategy echoes earlier two-flow architectures but integrates the motion signal directly into a transformer-augmented reconstruction framework.

At inference time, the system scores each frame using the reconstruction error, complemented by the peak signal-to-noise ratio, a standard image quality metric that drops when a reconstruction deviates sharply from the original. Frames whose reconstructions are poor, or whose PSNR falls below the pattern established by normal footage, are flagged as anomalous. The evaluation was conducted on four widely used benchmarks: UCSD Ped1 and UCSD Ped2, which capture pedestrian walkways with cyclists, skaters and occasional vehicles intruding into the frame; the Avenue dataset, filmed in a campus entrance hall with loitering, throwing and running; and ShanghaiTech, a large and challenging collection of thirteen scenes with diverse camera angles and crowd conditions.

The results are striking. On UCSD Ped1 the framework achieved an area under the receiver operating characteristic curve of 98.97 percent with an equal error rate of 4.09 percent, meaning the point where its false alarm rate and miss rate cross sits below five percent. On UCSD Ped2 it reached 95.63 percent AUC with a 4.34 percent EER, and on Avenue it recorded 93.53 percent AUC with a 12.23 percent EER. On the hardest benchmark, ShanghaiTech, it attained 89.14 percent AUC with a 16.49 percent EER. The consistent performance across datasets with very different scene dynamics, lighting conditions and anomaly types suggests the hybrid design generalizes rather than overfitting to a single environment, and the reported equal error rates on the UCSD benchmarks place the method among the strongest reconstruction-based approaches described in the literature.

The implications extend well beyond academic benchmarks. Security operators, transit authorities and smart-city planners all face the same economics: footage is cheap, attention is expensive. Systems that can reliably narrow a human operator’s focus to the small fraction of video that actually deserves scrutiny could change how surveillance is staffed and reviewed. Because the framework is unsupervised, it also sidesteps the privacy and labeling burdens of supervised training, since it requires only examples of ordinary activity, which every camera already records in abundance. The authors note that the datasets used in the study are available from the first author on reasonable request, and the work was carried out without dedicated funding.

Challenges remain before such systems can be trusted in the wild. Real deployments must cope with camera shake, weather, gradual shifts in what counts as normal as seasons and crowds change, and the ethical questions that accompany any technology capable of deciding, autonomously, what counts as suspicious behavior. The equal error rates on the more crowded and heterogeneous benchmarks, while strong, still leave room for false alarms that could erode operator trust. Yet the trajectory is clear. By marrying the local precision of convolutions, the global reasoning of transformers, a memory that remembers how normal scenes unfold in time, and an explicit motion signal from optical flow, this work offers a blueprint for surveillance AI that watches quietly, learns the rhythm of a place, and speaks up only when the rhythm breaks.

Subject of Research: Unsupervised spatio-temporal video anomaly detection using a memory-augmented CNN-ViT autoencoder with Farneback optical flow

Article Title: Spatio-temporal video anomaly detection via CNN-ViT autoencoder and farneback optical flow

Article References: Pathak, V., Diwakar, M., Pandey, N. K., Roka, S., & Singh, P. (2026). Spatio-temporal video anomaly detection via CNN-ViT autoencoder and farneback optical flow. Cluster Computing, 29(13), Article 773. https://doi.org/10.1007/s10586-026-06598-5

Image Credits: AI Generated

DOI: 10.1007/s10586-026-06598-5

Keywords: video anomaly detection, surveillance, autoencoder, vision transformer, CNN, optical flow, Farneback, memory module, unsupervised learning, computer vision, deep learning, UCSD Ped2