How to Reduce False Alarms in Distributed Systems with Interval Detection

Background

Monitoring highly distributed applications often involves hundreds of services across cloud and on‑premise environments, making error identification, latency detection, and root‑cause analysis difficult. Even with strong monitoring and alerting systems, infrastructure changes over time can cause unreliable anomaly detection, and 24/7 services rely on alerts for stability.

Developers frequently over‑monitor, receiving many false alerts that desensitize teams and allow real issues to slip through, leading to serious failures.

Alerts Are the Foundation of Reliability

Because perfect systems do not exist, we must continuously improve reliability. Effective alerts help us stay aware of service status and quickly locate problems.

Know the current state of services at all times.

Detect issues immediately and pinpoint their causes.

Alerts provide automated detection of abnormal conditions, serving as the primary means for teams to monitor service quality and availability.

Real‑World Alerting Problems

Dynamic business changes make static thresholds unsuitable

Metrics often exhibit hourly, daily, or weekly seasonality, so fixed thresholds generate many false alerts.

Different applications require different thresholds for the same metric

For example, a 200 ms response time may be normal for one API but 500 ms for a high‑traffic API, making a single static threshold impractical.

Thresholds evolve with business growth

As new services launch, metric baselines shift; without timely updates, false alerts increase.

Alert Setting Principles

To avoid interrupting work, alerts should be authentic, detailed, actionable, conservatively set initially, and continuously optimized.

Authenticity: alerts must reflect a real phenomenon.

Detail: describe the incident precisely.

Actionability: only notify when an operation is required.

Conservative thresholds: start broad to avoid missed alerts.

Continuous optimization: analyze and adjust to reduce false positives.

For example, a request‑failure alert that triggers on any failure may lack authenticity and actionability.

Alert Tool Selection

Grafana

Grafana supports many data sources, visualizations, and an alerting module that can configure rules directly from charts, providing friendly notifications.

However, Grafana’s alerts rely mainly on threshold comparisons and lack advanced outlier or change‑point detection.

Zabbix

Zabbix requires custom scripts for detection, with a more complex setup involving scenes, monitoring pages, and host triggers.

Its rules focus on expressions and thresholds, also lacking advanced outlier detection.

Open‑Falcon

Open‑Falcon offers flexible data collection, auto‑discovery, and high‑scale ingestion, but still relies on threshold‑based alerts and misses advanced detection features.

Observation Cloud Interval Detection

When static thresholds become insufficient, Observation Cloud computes a normal range using historical data and the Local Outlier Factor (LOF) algorithm, which combines distance and density factors to define anomalies.

The model samples points between the training set’s min and max, merges adjacent normal points, and forms one or multiple normal intervals, which are then used to suppress invalid alerts.

Alert Tool Usage

Utilize Observation Cloud’s monitor for interval detection.

Interval Detection Configuration

Basic Information

Rule name: name of the detection rule.

Associated dashboard: dashboard linked to the rule.

Detection Settings

Detection frequency: fixed intervals such as 5 min, 15 min, 30 min, 1 h.

Detection window: time range of metric data for each run.

Detection metric: only one metric per rule, must be a numeric series.

Trigger condition: defines alert levels.

Alert levels: urgent (red), important (orange), warning (yellow), no‑data (gray), normal (green). Each level has a single trigger condition.

Trigger condition: based on time range, abnormal count, direction, and proportion.

Alert Level Details

Urgent/Important/Warning: configure abnormal direction (up, down, both) and abnormal proportion.

Direction: whether data exceeds the upper bound, lower bound, or both.

Proportion: percentage of points outside the normal interval.

No‑data/Normal: detection period equals detection frequency; custom period = frequency × N. No‑data alerts can be configured to trigger, recover, or ignore.

Detection period = detection frequency.

Custom period = detection frequency × N.

No‑data: three handling options require manual configuration.

When a rule is active, the first no‑data detection does not generate an alert; subsequent continuous no‑data detections trigger a no‑data event.

Event Notification

Event title: name of the alert condition.

Event content: description, supporting template variables.

Alert strategy: defines which levels trigger notifications, recipients, and silence periods.

Applying Interval Detection to Disk Usage

Example: monitor host disk usage; when usage spikes beyond the normal interval, investigate the offending processes.

Detection configuration:

Detection window: recent 15 minutes of data.

Detection metric: host‑device memory usage percent.

Query example: disk:(AVG( used_percent)) BY host, device Trigger condition

Urgent (red): if more than 50 % of points in the last 15 minutes exceed the interval, trigger urgent alert.

Normal (green): if two consecutive cycles have no anomalies, consider the issue resolved.

No‑data (gray): if two consecutive cycles have no data, trigger a no‑data alert.

Trigger Events

After creating the monitor, a 5‑minute detection cycle caught a sudden memory usage spike, generating an alert.

Investigation revealed host izbp152ke14timzud0du15z had disk usage abnormal points covering 99.45 % of the interval.

Further analysis via the linked view identified a testing process that consumed excessive resources; terminating it restored normal operation.

Recovery Events

When ten consecutive cycles show no anomalies, the system automatically marks the issue as recovered.