When Large Models Begin to “Understand” Anomalies: The New Wave in Video Anomaly Detection

Traditional video anomaly detection (VAD) acted like a red‑light alarm: it could flag that something was off but could not explain why. Early methods trained on only normal videos, using reconstruction or prediction errors to assign an anomaly score. While this avoided the need for many abnormal samples, it suffered from false alarms caused by lighting changes, camera shake, and subtle yet dangerous semantic events that produced little pixel variation.

Weakly‑supervised approaches later introduced video‑level labels to locate suspicious clips, reducing annotation effort. However, they still relied on the scenes and anomaly categories seen during training, limiting their ability to handle truly unseen events.

Recent advances bring vision‑language models (VLM) and large language models (LLM) into VAD. Researchers first let a VLM generate a textual description of each frame, then feed the sequence of descriptions to an LLM that reasons over temporal context to estimate anomaly likelihood. The CVPR 2024 paper LAVAD exemplifies a training‑free pipeline: VLM‑generated captions are cleaned by cross‑modal similarity and refined by the LLM, turning raw pixel differences into semantic judgments such as “a person runs against the crowd” or “two people engage in a physical conflict.”

This shift yields three new capabilities:

From pixel deviation to semantic anomaly: subtle visual changes (e.g., an abandoned bag) acquire clear event semantics.

From closed categories to open vocabulary: the system can identify anomaly types never seen during training and answer “what is the anomaly?”

From detection to understanding: the CUVA benchmark adds the questions What, Why, and How, pushing research toward explanations of the anomaly’s cause and impact.

Looking forward, VAD is undergoing three transformations: cross‑scene generalization, verifiable explanations, and deployment in real‑time systems. Future evaluation will consider not only detection accuracy but also latency, energy consumption, privacy, false‑alarm cost, and human‑AI collaboration. The central challenge becomes whether a system can discover unseen dangers promptly, provide a verifiable rationale, and know when to defer judgment to a human operator.