How a Unified Frequency‑Domain Model Beats Custom Neural Nets for Cloud Anomaly Detection

Introduction

Recently, a paper titled “Learning Multi‑Pattern Normalities in the Frequency Domain for Efficient Time Series Anomaly Detection”, led by Alibaba Cloud’s big‑data engineering team together with Zhejiang University, was accepted by ICDE 2024. It addresses the problem that different cloud services exhibit distinct normal patterns, which traditional single‑model neural networks cannot capture effectively.

Background

Reconstruction‑based unsupervised anomaly detection models (e.g., OmniAnomaly, MSCRED, AnomalyTransformer, DCdetector) achieve high accuracy but each model only captures one or a few normal patterns. Meta‑learning approaches (e.g., PUAD, TranAD) still require separate models for each pattern, which is infeasible when tens of thousands of patterns exist in large‑scale cloud environments.

Challenges

A sample that is normal under one pattern may be anomalous under another, so a unified model must adapt to multiple standards.

Real‑time processing of massive monitoring data demands reduced inference latency and finer‑grained parallelism.

Existing reconstruction methods are insensitive to short‑term anomalies, yet cloud services exhibit both persistent and short‑term anomalies.

Breakthrough

Unlike methods that directly judge a sample as anomalous, MACE extracts anomalies from the relationship between a sample and the frequency‑domain subspace representing its normal pattern. It first projects the sample into the pattern’s frequency subspace; the reconstruction error grows with the distance to that subspace. Context‑aware Fourier transform/inverse transform leverages frequency sparsity for computational efficiency and enables fine‑grained, high‑concurrency implementation without temporal dependencies. Additionally, Peak Convolution and Valley Convolution enhance short‑term anomalies for easier detection.

Application

MACE has been integrated into Alibaba’s Feitian Big‑Data AI Control Platform (ABM) as an anomaly‑detection service, helping the platform discover anomalies promptly.

Paper title: Learning Multi‑Pattern Normalities in the Frequency Domain for Efficient Time Series Anomaly Detection

Authors: Chen Feiyi, Zhang Yingying, Qin Zhen, Fan Lunting, Jiang Renhe, Liang Yuxuan, Wen Qingsong, Deng Shuiguang

PDF: https://arxiv.org/abs/2311.16191