How Unified Alert Convergence Can Transform Monitoring Systems

Background

In the original Vivo monitoring 1.0 architecture each subsystem (basic, generic, tracing, log, probing) maintained its own metric calculation, storage, detection, and alert‑convergence logic. This siloed design prevented data fusion and limited the ability to support broader monitoring scenarios, prompting a redesign toward a unified monitoring platform that standardizes metric calculation, storage, detection, alerting, and presentation.

Current Situation

Previously each monitoring system performed its own alert convergence and message assembly before forwarding alerts to a legacy alert center, resulting in duplicated effort and inconsistent handling. The new design consolidates convergence, message assembly, and delivery into a single, loosely‑coupled service that can be adopted selectively by existing monitoring components.

Key Concepts

Exception : One or more metric values exceed a configured threshold within a detection window (e.g., a 6‑3 rule where three out of six points exceed the threshold).

Problem : A continuous period during which a collection of similar exceptions occurs; a problem may contain multiple exceptions.

Alert : Notification (SMS, phone, email, etc.) sent to users when a problem is reported.

Recovery : When all exceptions belonging to a problem no longer satisfy the detection rule, the problem is considered recovered and a recovery notice is emitted.

Metrics

Operational health is measured with standard incident‑management metrics. This summary focuses on MTTA (Mean Time To Acknowledge) and MTTR (Mean Time To Repair).

MTTA

MTTA is the average elapsed time from the occurrence of a problem to the moment the operations or development team acknowledges it.

t[i]

– timestamp when the i‑th service issue is responded to. r[i] – total number of occurrences of the i‑th service issue.

MTTR

MTTR is the average time required to restore a service to normal operation after an alert is raised.

t[ri]

– total time from the i‑th service’s r alerts to full recovery. r[i] – total number of alerts for the i‑th service.

Both metrics help identify bottlenecks in response and restoration processes.

Functional Design

Alert Convergence : Merges duplicate alerts using mechanisms such as first‑alert waiting time, configurable alert intervals, exception‑convergence dimensions, and message‑merge dimensions.

Alert Claim : Once an alert is claimed, subsequent identical alerts are routed only to the claimant, reducing duplicate handling.

Alert Mute : Suppresses alerts for known issues during defined periods (time‑based or event‑based).

Alert Callback : Invokes a user‑defined callback interface to trigger automated remediation actions.

Mis‑Alert Tagging : Allows operators to mark false alerts, feeding the information back into detection‑rule tuning.

Alert Escalation : Automatically escalates alerts that remain unaddressed beyond a configurable timeout, ensuring higher‑level attention.

Architecture

The unified alert service sits at the end of the monitoring pipeline and provides both alert delivery and business‑level notifications. It is decoupled from upstream monitoring services, allowing selective adoption of capabilities such as convergence, mute, or claim.

Core workflow:

Anomalies are consumed from Kafka topics or via a REST endpoint.

The convergence service deduplicates anomalies, creates a problem entity, and persists it in MySQL.

Problems are pushed into a Redis sorted‑set delay queue; the score represents the earliest expiration time.

A worker continuously polls the queue, extracts expired items, assembles the final message text according to configurable templates, and dispatches the alert through the selected channels (SMS, email, webhook, etc.).

Core Implementation

Alert convergence relies on a Redis‑based delay queue. Redis provides high‑performance sorted‑set ordering and persistence, guaranteeing reliable consumption even under failure scenarios.

When a problem passes deduplication, it is inserted into the delay queue with a score equal to its scheduled dispatch time. The queue orders items by the smallest score; a dedicated worker thread repeatedly checks for items whose score ≤ current timestamp, removes them, performs message templating (including placeholder substitution for dimensions such as ${name} or ${sex}), and sends the assembled alert via the configured transport.

Future Outlook

Further work should aim to reduce manual configuration, introduce AI‑assisted intelligent convergence, and strengthen root‑cause analysis. Integrating the unified alert service with upstream data sources and downstream incident‑response tools will streamline data flow, increase automation, and provide richer context for fault detection and remediation.

References

What are MTTR, MTBF, MTTF, and MTTA? A guide to Incident Management metrics – https://www.motadata.com/blog/incident-management-metrics/

Average Repair Time – (Chinese source)

Key Operations Metrics – http://blog.oneapm.com/apm-tech/289.html

PIGOSS TOC Smart Service Center – http://www.netistate.com/news1/452.html

Large‑Scale Intelligent Alert Convergence Practices – https://www.docin.com/p-2505292466.html

Can Redis implement a delay queue? – https://www.cnblogs.com/xiaowei123/p/13222710.html