Live Streaming Service Monitoring and Alert Attribution Practice

Background: With rapid growth of the DeWu community and live streaming services, user volume and stability requirements have increased. Quickly attributing monitoring alerts and resolving issues is essential.

This article shares a practice of consolidating alert‑attribution knowledge, encouraging team learning and case summarization to help engineers locate problems faster.

Scope: The focus is on attributing alerts to specific aspects rather than diagnosing particular business bugs. It discusses required logs, tracing, and context for code‑level issues.

Current stack: Services are written in Go, run on Kubernetes, metrics displayed in Grafana, and alerts sent via FeiShu. Alert rules include RT, QPS, goroutine, panic, HTTP status, and business exceptions.

Case study: A recent RT jitter was observed. Grafana showed traffic spikes and rising QPS, HTTP/GRPC latency, and MySQL/Redis latency.

Routing analysis: Logs revealed Redis timeout errors and third‑party call timeouts. System resources (CPU, memory) were normal, ruling out resource bottlenecks. Multiple pods across different nodes excluded single‑node failures and network issues.

Problem localization: Storage layers (MySQL, Redis) were checked via Alibaba Cloud RDS and showed no slow queries. Comparing with another service using the same storage confirmed storage health.

Attention turned to the service mesh (Istio). Initial Istio metrics seemed fine, but later it was discovered that the monitoring data were inaccurate. After fixing the data collection, Istio’s real load indicated overload caused by increased pod count and sidecar resource limits (2c1g). Reducing pod resources and increasing sidecar count mitigated the jitter.

Impact level & possible causes: The article lists potential CPU, memory, MySQL/Redis, and traffic‑path node issues, with a decision‑tree style funnel for narrowing down root causes.

Reference flow: Upon receiving an alert, determine impact scope, consider possible reasons, and eliminate them step by step, using cases such as “same storage, other services normal → storage not at fault”.

Traffic path & storage overview: Describes north‑south ingress, east‑west service‑mesh (Envoy), MySQL multi‑AZ high‑availability, and Redis proxy‑based HA architecture.

Conclusion: Fast alert attribution requires both code‑level knowledge and a solid understanding of system architecture.