Building Enterprise Alerting Systems with Alibaba Cloud Observability: A Practical Guide

1. Why Alerting Matters

Effective monitoring and alerting are indispensable for any continuously running production system; monitoring provides real‑time visibility while alerts enable rapid detection and response to abnormal conditions.

1.2 IT System Availability

According to ufried (2017) in "Resilient software design in a nutshell," system availability is measured by two metrics: Mean Time To Failure (MTTF) and Mean Time To Repair (MTTR).

1.2.1 Improving MTTF

Technical dimension: design high‑availability architectures, apply defensive programming, enforce coding standards, and conduct thorough testing (unit, integration, performance, chaos).

Mechanism dimension: adopt gray‑scale releases, observability, rollback capabilities, fault‑level mechanisms, and reward/punishment policies.

Cultural dimension: promote stability‑related policies, training, and examinations.

1.2.2 Reducing MTTR

MTTR can be decomposed as MTTR = MTTI + MTTK + MTTF + MTTV, where:

MTTI: Mean Time To Identify – time from fault occurrence to detection.

MTTK: Mean Time To Know – time from detection to confirmation.

MTTF: Mean Time To Fix – time from confirmation to actual fix.

MTTV: Mean Time To Verify – time from fix to verification.

Shortening MTTI and MTTK through robust alerting directly reduces overall MTTR.

2. General Alerting Process

Monitoring object inventory: identify all components and services across layers to ensure full coverage.

Metric analysis: determine which dimensions and indicators should be monitored for each object.

Metric collection: gather data from reliable sources (e.g., Alibaba Cloud CloudMonitor, SLS, ARMS) and clean/visualize it.

Alert rule configuration: set statistical thresholds, assign severity levels, and route notifications via DingTalk, SMS, voice, etc.

2.1 Monitoring Object Layers

Business layer (including terminal experience): core business logic such as shopping cart, order, payment.

Application layer: microservice processes, API endpoints.

Dependency layer (services/systems): downstream services and middleware (Kafka, RocketMQ, RDS, Redis, etc.).

Infrastructure layer: containers, hosts, networks, CPU, memory, disk, ports.

2.2 Metric Types

Flow metrics: request count, QPS, PV, throughput.

Error metrics: error count, error rate.

Latency metrics: response time, timeout count/rate.

Saturation metrics: utilization, capacity thresholds.

2.3 Multi‑Dimensional Analysis

Single‑dimensional thresholds often cause false alarms; combining time trends, business context, and resource utilization yields more precise detection. For example, QPS alone may trigger noise, but adding RT and business scenario reduces false positives.

3. Key Alerting Issues

Multi‑platform alerts: heterogeneous monitoring systems (Prometheus, Grafana, Zabbix) generate scattered alerts, making unified view and response difficult.

False‑positive alerts: alerts that do not correspond to real problems waste resources and cause “wolf‑howl” fatigue.

Alert storms: excessive alert volume overwhelms engineers; mitigation includes level segregation, filtering, compression, and silencing.

3.1 Alarm Levels (P1‑P4)

Based on Alibaba Group’s fault‑level definition, alerts are classified as:

P4: minor issue, no impact on customers.

P3: requires immediate attention, minor impact.

P2: serious issue affecting critical services.

P1: critical problem demanding immediate management intervention.

3.2 Notification Strategies

Voice: only for core P1 alerts.

SMS: for core P1‑P2 alerts.

IM (DingTalk) messages: for all core alerts and non‑core alerts routed to groups.

IM groups: used for core alerts and multi‑person coordination.

4. Case Study: Enterprise A

Enterprise A, a large vertical e‑commerce company, faced insufficient monitoring coverage, unclear alert thresholds, and fragmented alert ownership across >30 vendors.

4.1 Design & Implementation

Inventory existing monitoring assets and define a layered object model.

Standardize tagging across all resources; enforce tagging via daily scripts and SOPs.

Integrate Alibaba Cloud observability products (ARMS, CloudMonitor, SLS) and supplement missing metrics with custom solutions.

Use ITSM to distribute alert policies.

Build a unified alert dashboard.

4.2 Tagging & Standards

Adopt the tagging best‑practice guide (see reference [3]) and enforce log standards (mask sensitive data, include trace IDs, log levels). Tags are applied via console, TagResources API, or YAML for containers.

4.3 Metric & Alert Configuration

CloudMonitor alerts for ECS, RDS, etc.

ARMS application monitoring and browser monitoring alerts.

Prometheus alerts with custom PromQL, e.g.:

sum(container_memory_working_set_bytes{id!="/",container!=""}) BY (instance, name,container, pod_name , namespace) / sum(container_spec_memory_limit_bytes{id!="/"} > 0) BY (instance, name, container, pod_name , namespace) * 100) > 85

SLS log‑based alerts per logstore.

4.4 Alert Enrichment & Distribution

Enrich ARMS alerts with enrich_app_labels:true so that custom application and host tags are attached. Use Alibaba Cloud tags and Kubernetes labels to route alerts to the appropriate owners.

4.5 ITSM Integration

Integrate CloudMonitor and SLS alerts into ARMS via webhook callbacks, then configure ARMS notification policies (voice, SMS, IM) according to the P‑level definitions.

4.6 Unified Dashboard

A consolidated dashboard displays alerts from CloudMonitor, SLS, and ARMS, enabling cross‑vendor comparison, SLO tracking, and root‑cause analysis.

5. Conclusion

The guide presents a complete methodology for constructing an enterprise‑grade alerting system: define availability goals (MTTF/MTTR), inventory objects, select and instrument metrics, apply multi‑dimensional analysis, establish realistic alert thresholds, classify alerts by severity, choose appropriate notification channels, and continuously refine the system through PDCA cycles. The real‑world case demonstrates how these practices can be applied using Alibaba Cloud observability services to achieve reliable, scalable, and manageable alerting.