How Machine Learning Transforms Database Monitoring: From Fixed Thresholds to Intelligent Anomaly Detection

Why Traditional Inspection Is Not Enough

Conventional monitoring systems rely on static thresholds that are deliberately set high to avoid flooding alerts. While they can catch obvious spikes, they miss gradual changes that occur as traffic grows, such as increasing QPS or data volume during peak periods, leading to delayed response and potential service disruption.

Anomaly Detection: The "Second Pair of Eyes" for Inspection

Recent advances in machine learning and AI enable time‑series monitoring to learn normal patterns from historical data and flag deviations that fall outside those patterns. The Qunar DBA team applies this approach to daily inspections, using intelligent analysis to capture dynamic changes in database metrics.

1. Feature Extraction

Metrics with periodic behavior (e.g., Redis memory usage that drops at night and rises in the morning) must be identified and excluded from anomaly detection. Conversely, metrics that show a steady upward trend (e.g., MySQL disk usage) should be monitored for sudden large increases. The first step extracts periodicity, stability, and growth characteristics for each instance.

2. Algorithm Selection

Depending on the extracted features, different transformation and detection algorithms are applied:

Sliding‑window transformation creates a new series by subtracting the average of a left window from a right window, optionally using median, max, or IQR.

Seasonal detection uses the SeasonalAD pipeline to remove seasonal components via classic decomposition and then applies IQR or sign checks.

Level‑shift detection ( LevelShiftAD) captures sudden spikes in CPU or memory watermarks.

3. Tuning

Purely mathematical anomaly points may not correspond to real incidents. The system combines dynamic thresholds learned from historical data with static thresholds to filter out false positives, such as DDL operations, data migrations, or low‑usage scenarios that would otherwise trigger alerts.

4. Alert Convergence

To avoid alert storms, the platform implements both vertical and horizontal convergence:

Vertical convergence groups repeated alerts for the same issue within a configurable time window, updating the message with the latest details.

Horizontal convergence merges alerts from multiple instances in the same cluster or across clusters belonging to the same business group, reducing duplicate notifications.

Three Typical Application Scenarios

1. Stable‑Trend Anomaly Detection

Most metrics (MySQL disk usage, server memory, Redis memory) exhibit a stable or slowly rising trend under normal conditions. The DoubleRollingAggregate algorithm creates a differential series; positive values indicate upward movement, negative values indicate decline. By measuring the proportion of points above or below zero, the system decides whether the overall trend is abnormal.

Implementation steps include filtering hosts with low disk usage, checking growth beyond a configured threshold, and excluding known interference such as DDL work or data migration.

2. Periodic‑Change Anomaly Detection

Metrics like Redis memory often follow a daily cycle. The system first verifies that a metric has a periodic pattern, then uses SeasonalAD to remove the seasonal component and applies both sign checks and IQR on the residuals. Only points that break the periodic pattern are reported.

3. Sudden‑Shift (Burst) Anomaly Detection

This scenario captures rapid spikes or drops within a short time window. Examples include a sudden CPU rise on a MySQL server or a QPS drop to near zero. The LevelShiftAD pipeline combines a double‑rolling aggregate (absolute difference) with IQR or sign checks to pinpoint the exact moment of change.

Additional tuning filters out bursts that are part of known periodic behavior or that occur when the underlying metric is already low, preventing unnecessary alerts.

Practical Experience and Optimization

To date, the system monitors dozens of key metrics (CPU, disk, memory, table size, QPS, scan rows) with an alert accuracy above 80 %. Different metrics exhibit distinct data characteristics, so a single algorithm cannot cover all cases. The team groups metrics by similar mathematical features and assigns a suitable algorithm suite, achieving roughly 80 % coverage with generic logic and handling the remainder with custom rules.

Dynamic thresholds learned by machine learning adapt to evolving patterns, while static thresholds provide a safety net for extreme cases. Multi‑dimensional alert convergence dramatically reduces noise, making alerts more actionable for DBAs.

Future Outlook

The roadmap moves from single‑metric detection toward multi‑metric correlation and automated root‑cause analysis. By progressively adding joint analysis of related metrics, the system aims to increase coverage and precision, eventually handling more complex diagnostic tasks and freeing DBAs to focus on high‑value optimization and incident response.