How Goldeneye Enables Adaptive, Intelligent Business Monitoring at Scale

Background

Goldeneye is Alibaba Mom's business monitoring platform that builds on real‑time log collection and statistical analysis to provide alerts and assist in fault localization. Existing internal monitoring platforms are open and low‑cost but require users to set thresholds manually, leading to high maintenance effort.

Technical Background

Typical monitoring systems consist of collection, data processing, detection, and alarm modules. Goldeneye uses Alibaba's internal middleware: Time Tunnel agents pull logs to a Topic and store offline logs in ODPS; jstorm and ODPS MR jobs handle real‑time and batch processing; results are stored in HBase. The presentation focuses on threshold prediction, detection, alarm generation, and assisted localization.

Key Ideas

Intelligent monitoring replaces manual threshold setting with data‑driven prediction. By analyzing historical samples, the system predicts baseline values and adaptive thresholds, then uses rule combinations and mean‑shift algorithms to detect anomalies.

Adaptive Thresholds

Traditional static thresholds (fixed lines or relative change percentages) cause false positives or missed alarms and require constant manual tuning. Goldeneye automatically adjusts thresholds based on predicted values, similar to an automatic transmission that changes gears according to speed.

Automatic Monitoring Item Discovery

When the system can predict dynamic thresholds, it can also automate the addition, removal, and adjustment of monitoring items. This avoids the labor‑intensive process of manually tracking thousands of metrics that frequently change.

Balancing False Positives and Missed Alarms

Goldeneye first flags suspicious points with a stricter dynamic‑threshold detector, then filters them using predefined combination rules and convergence expressions (e.g., only alert when metrics M1 and M2 are abnormal together). This strategy keeps missed alarms low while reducing false alerts.

Implementation Details

The system ingests four inputs: real‑time data, historical data, prediction strategy, and alarm‑filter rules.

Threshold parameters: coefficient‑based upper/lower bounds, time‑segment prediction coefficients, sensitivity coefficients.

Prediction parameters: sample size, Gaussian filter water‑level or filter ratio, confidence from mean‑shift segmentation.

Alarm convergence: define which repetitions trigger notifications, merge multiple instance alarms, and set custom convergence expressions.

Dynamic‑threshold prediction steps:

Sample selection – typically the past ~50 days, separating workdays and holidays.

Outlier removal – filter samples with Gaussian probability < 0.01 or absolute sigma > 1.

Segment selection – apply a mean‑shift model to split the time series; keep the most stable recent segment.

Baseline prediction – use exponential smoothing on the cleaned, ordered samples, then compute upper/lower bounds using sensitivity or coefficient settings.

Change‑point detection uses a mean‑shift (CUSUM) model. The algorithm converts subtle downward trends into a CUSUM series that rises on the left side of a change point and falls on the right, making the shift visually obvious. The first‑order derivative of the CUSUM series determines the exact change point, with iteration count and confidence adjustable by the user.

Intelligent Panorama

Combining adaptive thresholds with change‑point detection achieves a balance of low miss‑rate and low false‑alarm rate without ongoing manual maintenance, enabling unlimited monitoring coverage. Automatic discovery rules (e.g., "metric M > X ⇒ monitor") can assign different sensitivities to core versus peripheral dimensions.

Assisted Root‑Cause Localization

Alarms aim to reduce loss, so Goldeneye provides programmable assistance such as:

Full‑link tracing across services to identify upstream issues.

Correlation of alarm timestamps with operational events.

A/B testing or Top‑N analysis on dimension values to narrow down problematic segments.

Statistical correlation (e.g., Pearson) between related metrics, with manual configuration currently.

These techniques rely on a well‑structured metric taxonomy and metadata management system.