How NetEase Cloud Communication Builds a Real-Time Service Monitoring Platform

Data is essential for many businesses, and NetEase Cloud Communication uses data to improve services and drive continuous growth. Their service monitoring platform acts like a dashboard for a high‑performance car, showing speed, fuel, and RPM to help decide when to accelerate or brake, thereby ensuring stable, reliable communication services.

System Architecture

The platform processes audio‑video data primarily from client and server logs. The overall data collection chain is critical for data validity and timeliness.

Data Collection

Data enters the platform via two channels: an HTTP API for near‑real‑time reporting and Kafka for high‑throughput scenarios. A preprocessing module filters illegal data and performs early splitting before forwarding to the processing service.

Data Processing

After collection, data passes through task scheduling, consumer queues, and processing units. The platform supports generic rules (JSON conversion, field extraction) for 80% of cases and custom scripts for complex calculations such as multi‑field correlation, regex, and stream joins. Dimension tables handle high‑volume, high‑concurrency scenarios using local and third‑party caches. Processed data is written to NTSDB, a clustered time‑series database based on InfluxDB, offering high availability, compression, and concurrency.

Monitoring & Alerting

The alerting stage consists of a metric aggregation module and an alert module. Aggregation supports field grouping, flexible windows, filtering, and a rich set of operators (sum, min/max, first/last, avg, count, distinct, TP90/TP95/TP99, ring‑ratio, std‑dev) as well as composite metrics. To mitigate hotspot skew, a two‑stage aggregation is used: a pre‑processing shuffle followed by final aggregation.

Alerts are decoupled from aggregation, performing rule validation, rate limiting, message packaging, and delivery via internal IM, SMS, or phone channels.

Data Applications

Processed data feeds several downstream platforms:

Visualization : Grafana visualizes metrics stored in NTSDB; custom plugins extend Grafana for specialized dashboards.

Quality Service Platform : Provides real‑time problem‑locating tools for customers.

ELK Log Platform : Uses Logstash, Elasticsearch, and Kibana for detailed log search.

Online/Offline Analysis : Kafka streams data to Flink for slicing and archiving, then syncs to an offline warehouse for further mining.

Conclusion

Since its launch in early 2020, the platform has grown from a dozen collection tasks to over 300, handling 100+ key user behaviors and system events, 300+ core audio‑video metrics, processing millions of rows daily and terabytes of data. Continuous growth in concurrency and throughput drives higher demands for stability and scalability, which the platform strives to meet while delivering higher‑quality services to customers.