How Tencent's Intelligent Monitoring Transforms Ops Automation

Monitoring Significance

Operations automation is a topic of efficiency improvement. The speaker emphasizes that true value for ops comes from quality assurance—reliability, availability, and user experience.

Reliability is measured through monitoring, which informs availability assessments and optimization suggestions. Improving these metrics directly enhances user experience.

Monitoring Methods

Three main approaches are used:

Active – developers embed instrumentation before release to meet ops requirements.

Passive – external probes (e.g., ping) detect issues without relying on application reporting.

Side‑channel – third‑party data such as sentiment analysis complements active and passive monitoring.

Monitoring Essence

All monitoring points can be reduced to three key indicators: request volume, success rate, and latency. Trend analysis, clustering, and other data‑processing strategies highlight the most critical issues for engineers.

Monitoring System Goals – Full, Fast, Accurate

The goal is to achieve comprehensive coverage without blind spots, while delivering alerts quickly and precisely, minimizing false alarms.

Full‑Link Monitoring

Mobile internet introduces diverse access methods, carrier variations, and fragmented client versions. Comprehensive monitoring requires numerous points across network, server, and infrastructure layers, including specialized points for mobile scenarios such as slow‑analysis and sentiment monitoring.

Monitoring Speed

A waterfall diagram illustrates the latency from data collection to alert delivery. Optimizations focus on reducing processing time while maintaining cost‑effectiveness.

Unified Reporting Protocol

Data is categorized as three‑dimensional (ID, timestamp, value) or multi‑dimensional (service‑specific contexts). This classification enables faster alert generation and efficient storage.

Intelligent Monitoring – ROOT System

With billions of daily alerts, traditional convergence methods are insufficient. The ROOT system uses topology, time correlation, and weight‑area algorithms to filter noise, identify root causes, and present concise alerts.

Dimensionality Reduction

Automatic topology generation leverages routing components and service‑call data. TCP/UDP packet analysis further refines relationships between services.

Time Correlation Analysis

Long‑standing red alerts are filtered out as low‑priority, focusing on recent anomalies that impact user experience.

Weight‑Area Analysis

Algorithms assign higher weight to downstream modules and linked alerts, helping prioritize root‑cause investigation.

Quality Ecosystem

Monitoring alone cannot solve all problems; a collaborative ecosystem involving development, ops, QA, product, and management is essential for continuous improvement.

Sky‑Net Classification

Monitoring points are classified by business layer responsibility, defining a DLP (critical point) for each module. Alerts are routed to appropriate teams via differentiated channels (QQ, SMS, WeChat, phone) based on severity.

Q&A

Active, Passive, Side‑channel Proportions

Proportions vary by scenario; side‑channel is least common but most indicative of user impact.

Can Alerts Trigger Self‑Healing?

High‑precision alerts enable classification and automated remediation.

Self‑Healing Rate Metrics

Basic alerts are fully automated; business‑level alerts are progressing toward higher self‑healing rates.

Ensuring Convergence Doesn't Drop Useful Alerts

Converged alerts are prioritized for ops, while raw monitoring data remains accessible to developers for deeper analysis.