How Unified Alert Convergence Can Slash Monitoring Noise and Boost MTTA/MTTR

Background

In the original Vivo monitoring 1.0, each sub‑system (basic monitoring, general monitoring, tracing, log monitoring, synthetic monitoring) maintained its own detection, aggregation and alert‑convergence logic. The subsystems first performed local convergence and then forwarded alerts to an old alert centre, resulting in duplicated rule maintenance despite high functional overlap. This siloed architecture prevented data fusion and limited the ability to support broader monitoring scenarios, motivating a redesign toward a unified monitoring service.

Core Concepts

Exception : Within a configurable detection window, one or more metric points exceed a threshold, generating an exception. Example: a 6‑3 rule (window size = 6, at least 3 points > 95) produces an exception only in the second window where three points cross the threshold.

Problem : A continuous series of similar exceptions is grouped into a single problem entity. Multiple exceptions can map to the same problem (one‑to‑many).

Alert : A notification (SMS, phone, email, etc.) sent to users when a problem is raised.

Recovery : When all exceptions belonging to a problem no longer satisfy the detection rule, the problem is considered recovered and a recovery notification is emitted.

Operational Metrics

Following Peter Drucker’s principle, system health is measured by metrics such as MTTD, MTTA, MTTF, MTTR and MTBF. The design focuses on MTTA (Mean Time To Acknowledge) and MTTR (Mean Time To Repair).

MTTA is calculated as:

t[i] – time from the i‑th service encountering a problem to the moment the ops or dev team acknowledges it.

r[i] – total number of problem occurrences for the i‑th service.

MTTA reflects the responsiveness of the operations/dev team and the efficiency of the alert pipeline (source [1]).

MTTR is calculated as:

t[ri] – total time from the i‑th alarm occurrence to full service recovery.

r[i] – total number of alarms for the i‑th service.

MTTR captures the average time required to restore normal service, including detection, diagnosis and any necessary testing (source [2]). Three MTTR variants are distinguished:

Mean time to recovery – from alarm to restored service.

Mean time to respond – from first alarm to the start of remediation (excluding alert‑system latency).

Mean time to resolve – from detection through root‑cause elimination and verification.

Alert‑Storm Problem

When thousands of alerts are generated simultaneously, operators experience alert fatigue, leading to missed critical events (source [4]). Reducing alert volume while preserving essential information is therefore a primary objective.

Design Goals

Analysis of MTTA/MTTR drives three design dimensions:

Alert quantity – limit the number of notifications sent.

Alert convergence – merge co‑occurring exceptions into a single alert.

Alert escalation – automatically raise the priority of un‑acknowledged problems.

Functional Mechanisms

Key functions and their concrete behaviours are:

First‑Alert Wait : After an exception is generated, the system delays the first alert for a configurable period (e.g., 5 s). If another exception of the same problem arrives within the wait window, both are merged. Example: node 1 and node 2 of service A raise exceptions within 5 s → a single alert is sent.

Alert Interval : While a problem remains unresolved, the system re‑sends the alert at a configurable interval to keep stakeholders informed without flooding them.

Exception Convergence Dimension : Defines which attributes (e.g., node path) are used to group exceptions. Exceptions sharing the same dimension are merged before alert generation.

Message Merge Dimension : Specifies which fields are retained in the final alert text. The article illustrates this with a placeholder example where ${sex} is merged by dimension (single value) and ${name} is concatenated for all matching exceptions.

Alert Claim : When an operator claims an alert, subsequent identical alerts are routed exclusively to the claimant, reducing duplicate handling.

Alert Silencing : Allows a known problem to be muted for a defined window (e.g., during a release), preventing unnecessary notifications.

Alert Callback : Configurable callback API is invoked on alert generation to attempt automatic remediation.

False‑Alert Tagging : Users can mark an alert as false; the tag feeds back into detection‑model tuning.

Alert Escalation : If a problem remains un‑acknowledged beyond a timeout, the system automatically escalates to higher‑level personnel.

Unified Alert Architecture

The unified alert service sits at the end of the monitoring pipeline, providing both alert delivery and generic notification capabilities while remaining decoupled from upstream monitoring services.

Core processing flow:

Exceptions are ingested either from Kafka topics or via a RESTful API.

An exception handler creates a problem entity, persisting both the problem and the raw exception in MySQL.

The convergence module pushes the problem into a Redis delayed queue. The queue uses a sorted‑set where the score represents the intended release timestamp.

A watcher continuously polls the sorted‑set, extracts the smallest‑score (earliest‑expiry) tasks, assembles the final message (including placeholder substitution), and dispatches the alert through configurable channels (SMS, email, webhook, etc.).

Supporting services include a configuration‑management service for alert‑rule definitions and a metadata‑sync service that supplies auxiliary data (e.g., node topology) required for convergence.

Implementation Details

Redis is chosen for the delayed queue because its sorted‑set offers high‑performance score ordering and persistence. The queue aggregates exceptions belonging to the same problem within the wait window, thereby reducing duplicate alerts. For instance, three nodes of service A generate simultaneous exceptions; after de‑duplication they are merged into a single alert.

Processing steps (illustrated in the delayed‑task diagram):

Before enqueuing, the system checks whether a problem with the same key already exists to avoid duplicate entries.

The problem’s release time is encoded as the Redis score; the smallest score is always at the head of the queue.

A listener extracts expired tasks, performs message assembly (replacing placeholders such as ${sex} and ${name}), and forwards the assembled alert via the configured channel.

Future Outlook

Short‑term priorities are to tighten the data pipeline, automate configuration, and enrich alert dimensions for finer‑grained incident handling. Longer‑term work aims to incorporate AI‑driven (AIOps) techniques for intelligent convergence and root‑cause analysis, although large‑scale adoption remains pending (see references [5] and [6]).

References

[1] "What are MTTR, MTBF, MTTF, and MTTA? A guide to Incident Management metrics".

[2] "平均修复时间" (Mean Time To Repair) – technical blog.

[3] "运维不容错过的4个关键指标！" – discussion of MTTA improvement.

[4] "PIGOSS TOC 智慧服务中心让告警管理更智能" – analysis of alert‑storm impact.

[5] "大规模智能告警收敛与告警根因技术实践" – case study on large‑scale alert convergence.

[6] "你知道Redis可以实现延迟队列吗?" – explanation of Redis delayed‑queue implementation.