How Spatio‑Temporal Autoencoders Detect Anomalies in Real‑World Traffic Video

Abstract

Detecting abnormal events in real‑world video streams is challenging because anomalies are diverse and scenes contain cluttered backgrounds, objects, and motions. Existing methods rely on handcrafted features in local spatial regions. Inspired by recent advances in action recognition, we propose a spatio‑temporal autoencoder (STAE) that learns video representations with 3D convolutions and introduces a weight‑decaying prediction loss. Experiments on public datasets and a new traffic surveillance dataset show that our method significantly outperforms the previous state‑of‑the‑art.

1 Introduction

Automatic detection of anomalous events in video streams is a fundamental problem for intelligent video surveillance. Unlike supervised tasks such as action recognition, video anomaly detection suffers from severe class imbalance and high intra‑class variation of normal events, making supervised approaches infeasible. Unsupervised learning of normal video patterns and treating deviations as anomalies is the common solution.

Hand‑crafted features (e.g., HOG, HOF, 3D gradients) have limited representational power for complex traffic scenes. Deep learning, especially autoencoders, can learn discriminative features, but prior works use only 2D convolutions or fully‑connected layers, ignoring temporal cues.

Motivated by the success of 3D convolutional networks, we design a spatio‑temporal autoencoder that applies 3D convolutions in the encoder and 3D deconvolutions in the decoder, enabling the model to capture motion patterns across time.

In addition to the reconstruction loss, we introduce a weight‑decaying prediction loss that forces the network to predict future frames, encouraging the encoder to learn temporal dynamics. The model yields a regularity score per frame, which drops sharply at anomalous events (see Figure 1).

Most real‑world anomalies are complex, and existing datasets contain only synthetic or appearance‑based anomalies. To evaluate practicality, we collected a challenging traffic surveillance dataset composed of real‑world videos. Experiments demonstrate that our model works well on this dataset.

Our main contributions are:

We propose the first video anomaly detection model based on 3D convolutions, the spatio‑temporal autoencoder.

We introduce a weight‑decaying prediction loss that improves abnormal event detection.

We release a new traffic surveillance dataset and show that our method outperforms previous best results on both public benchmarks and the new dataset.

2 Our Method

We first review 3D convolutions and then describe the STAE architecture.

2.1 3D Convolution

Standard 2D CNNs process spatial dimensions only. 3D convolutions extend kernels over time and space, allowing simultaneous extraction of temporal and spatial features.

2.2 3D Convolutional Autoencoder

Input data. Unlike image classification where inputs are single RGB frames, anomaly detection requires a clip of multiple consecutive frames. We construct a 4‑D tensor by stacking T frames along the temporal dimension and apply 3D convolutions.

Data augmentation. Random cropping, brightness changes, and Gaussian blur are applied to sampled clips to increase training data while preserving motion speed.

Network architecture. The encoder consists of stacked 3D convolutional layers. The decoder splits into two branches: one reconstructs past frames, the other predicts future frames.

2.3 Weight‑Decaying Prediction Loss

We add a prediction branch that forecasts the next T frames. To avoid penalizing the model for unpredictable new objects, we apply a decaying weight to the loss of each predicted frame, emphasizing short‑term motion.

2.4 Regularity Score

Normal video clips obtain high regularity scores (low reconstruction error), while anomalous clips receive low scores, enabling anomaly localization.

3 Experiments

3.1 Datasets

We evaluate on UCSD Pedestrian, CUHK Avenue, and a newly collected Traffic dataset.

3.2 Visualizing Anomalies

Regularity scores derived from reconstruction error drop at abnormal frames. Figure 4 shows examples of error maps highlighting anomalous regions.

3.3 Anomaly Detection

Regularity score curves (Figure 5) demonstrate clear drops during anomalies across all datasets.

Tables 2 and 3 compare our method with recent approaches, showing superior performance on both public and traffic datasets.

3.4 Future Frame Prediction

The prediction branch can accurately forecast the motion of existing vehicles (green boxes) but fails to predict newly appearing vehicles (red boxes), as illustrated in Figure 6.

4 Conclusion

Future work includes exploring alternative network architectures, fusing multimodal inputs (e.g., RGB and optical flow), evaluating regularity scores at the instance level, and applying the framework to more complex scenarios.