Unlocking Smart Anomaly Detection in Alibaba Cloud Prometheus

Background

Anomaly detection is a core function of AIOps systems that automatically discovers abnormal fluctuations in KPI time‑series, providing the basis for alerts, auto‑mitigation, and root‑cause analysis.

What is anomaly detection?

Anomaly detection identifies abnormal events in time‑series data, such as unexpected rises, drops, or irregular oscillations.

Time‑series definition

A time series is an ordered set of data points collected at regular intervals (e.g., every minute or five minutes).

Limitations of open‑source Prometheus

The open‑source Prometheus version relies on manually configured static thresholds, which introduces several problems.

Common challenges

Configuring reasonable thresholds for tens of thousands of metrics is time‑consuming and often subjective.

Static thresholds become outdated as business metrics evolve, leading to high maintenance costs.

Poor data quality—such as latency, missing points, or noisy spikes—makes static rules ineffective.

Alibaba Cloud Prometheus extensions

Template‑based detection rules

For well‑understood scenarios, users can select a ready‑made template (e.g., "CPU usage > 80%") without manually setting thresholds.

Smart detection operator

The operator uses historical data length as a parameter, allowing the model to automatically follow metric trends and reduce the need for periodic manual reviews. It can also interpolate missing points (linear, polynomial, etc.) and automatically select the best model for noisy series.

Business value 1 – Efficient, high‑quality alert configuration

Template rules provide quick, consistent alert definitions for stable metrics, eliminating manual threshold tuning.

Business value 2 – Adaptive tracking of business changes

The smart operator self‑adjusts to evolving metric trends, lowering maintenance overhead.

Business value 3 – Robust to low‑quality data

Interpolation handles missing values, and model selection mitigates false alerts caused by spikes or noise.

Scenario examples

Spike/Drop metrics (e.g., QPS)

Static thresholds struggle with sudden spikes or drops. The smart operator adapts to these changes and correctly identifies abnormal surges or declines.

Periodic metrics

The system extracts period and offset values, removes periodic and trend components, and performs anomaly detection on the residual. This captures anomalies that static thresholds miss.

Trend‑breaking metrics

Metrics that normally rise or fall may experience sudden trend breaks. The smart operator selects the optimal model for such cases, detecting deviations from the overall trend.

Best‑practice workflow

Log in to ARMS‑Prometheus or Grafana and select the Prometheus data source.

Choose the metric you want to monitor.

Enter the anomaly detection operator in the query editor.

Operator definition

"anomaly_detect": {
  Name: "anomaly_detect",
  ArgTypes: []ValueType{ValueTypeMatrix, ValueTypeScalar},
  ReturnType: ValueTypeVector,
}
Input: metric time‑series (range vector) and detection parameter (default 3)
Output: 1 for anomaly, 0 for normal

Usage example

anomaly_detect(node_memory_free_bytes[20m],3)

The input must be a range vector, so a time window such as [180m] is required. If the metric has been aggregated, use [180m:] to keep it as a range vector, e.g., anomaly_detect(sum(node_memory_free_bytes)[180m:],3).