Detecting Time‑Series Anomalies with the Anomaly Transformer’s Association Discrepancy

Introduction

Real‑world systems generate massive continuous time‑series data. Detecting anomalous points is essential for safety and cost avoidance. The paper “Anomaly Transformer: Time Series Anomaly Detection with Association Discrepancy” proposes a criterion that distinguishes normal from abnormal points by comparing two types of associations for each time step.

Anomaly Judgment

For a given time step t, prior‑association measures similarity to its neighboring points, while series‑association measures similarity to all points in the series via standard Transformer attention. When an anomaly occurs, nearby points are also likely anomalous, so both associations focus locally and become similar, yielding a small discrepancy. In normal regions the series‑association spreads globally, producing a larger discrepancy. The discrepancy is quantified as the mean Kullback‑Leibler (KL) divergence between the two association distributions and serves as an anomaly score.

Prior‑association is modeled with a learnable Gaussian kernel K_σ(i,j)=exp(-(i-j)^2/(2σ^2)) whose scale σ is learned. Series‑association is obtained from the softmax attention weights of a Transformer encoder. The loss term for association discrepancy is

ℒ_KL = (1/T) * Σ_{t=1}^{T} KL(P^{prior}_t || P^{series}_t)

where P^{prior}_t and P^{series}_t are the normalized prior and series attention vectors for time t.

Anomaly Transformer Architecture

The overall architecture follows the standard Transformer encoder stack, but replaces each self‑attention block with an Anomaly‑Attention block that simultaneously computes prior‑ and series‑associations. Directly minimizing ℒ_KL would drive σ → 0, collapsing the Gaussian kernel. To prevent this, a Minimax training strategy is introduced.

Minimax Training Strategy

The training alternates between two phases:

Minimize phase : The prior‑association is encouraged to approximate the series‑association learned from the raw sequence, allowing the Gaussian kernel to adapt to diverse temporal patterns.

Maximize phase : The series‑association is optimized to enlarge the KL discrepancy, forcing attention to attend to non‑adjacent horizons.

Unsupervised representation learning uses a reconstruction loss ℒ_rec (e.g., mean‑squared error between input and decoder output). The total objective combines reconstruction and discrepancy terms:

ℒ = ℒ_rec + λ·ℒ_KL

λ

balances reconstruction and anomaly‑specific objectives.

Experimental Evaluation

The model was evaluated on five benchmark datasets covering service monitoring, aerospace telemetry, and other domains. Across all datasets the Anomaly Transformer achieved state‑of‑the‑art performance, outperforming prior methods on standard metrics such as precision, recall, and F1‑score.

Ablation studies compared (i) the full Minimax strategy versus training without the maximize phase, (ii) using a fixed Gaussian kernel versus the learnable prior, and (iii) removing the KL‑based anomaly criterion. Each ablation reduced performance, confirming the contribution of the training schedule, the adaptive prior, and the association‑discrepancy score.

References

Paper: https://arxiv.org/abs/2110.02642

Code repository: https://github.com/thuml/Anomaly-Transformer