How Alibaba Scales Anomaly Detection Across Millions of Metrics

Background

When a cluster reaches a scale of 100,000 machines and each machine reports over 100 metrics, the monitoring system must handle tens of millions of data points, making fast anomaly detection a critical challenge.

Typical problems include:

Abnormal machines are hard to spot.

The sheer number of metrics exceeds monitoring capacity.

Gray‑released machines may show hidden anomalies.

Construction Idea

Two main approaches are considered for anomaly detection:

Vertical comparison (time‑series) : Detect anomalies by comparing a metric against its historical values using time‑series algorithms.

Horizontal comparison (clustering) : Compare the same metric across different machines over a time window, using clustering techniques to spot outliers.

Choosing the right method involves evaluating scalability, timeliness, accuracy, and manual effort.

Horizontal clustering scores better on scalability and manual effort, while vertical comparison offers higher timeliness.

Technical Practice

The team selected the DBSCAN clustering algorithm because it can group metrics and identify outlier machine groups.

Implementation flow:

Data ETL: Use Alibaba Cloud MaxCompute to perform hourly and daily ETL on metric data.

Data Computation: Run DBSCAN with Python’s sklearn library.

Data Presentation: Visualize results on the business portal.

Visualization components include a PCA‑reduced 2‑D plot, a radar chart for cluster metric differences, and detailed cluster summaries to pinpoint problematic machines or metrics.

Application Effect

The solution uncovered high‑load machines that traditional monitoring missed, such as a subset of an online HDFS cluster where DataNodes were registered but not receiving data.

Conclusion

Horizontal clustering efficiently identifies machines with abnormal behavior, complementing vertical time‑series analysis. Combining both methods provides a more robust anomaly detection framework for large‑scale operations.