How to Quickly Attribute Live‑Streaming Alert Issues in a Kubernetes Environment

Background

With the rapid growth of the community and live‑streaming business, user volume has surged and service stability requirements have become increasingly stringent. The article focuses on how to quickly attribute monitoring alerts to their root causes, enabling faster problem resolution for both experienced engineers and those with less troubleshooting experience.

Scope of the Practice

The discussion does not dive into specific business‑level bugs; instead, it explains how to map an alert to a particular layer (e.g., resource, storage, traffic path) using comprehensive logs, full‑link tracing, and contextual information. The same reasoning applies to code‑level issues.

Technical Environment

The live‑streaming service runs Go applications in a Kubernetes cluster. Metrics are visualized in Grafana, and alerts are sent via Feishu. Existing alert rules include RT anomalies, QPS spikes, goroutine growth, panic events, HTTP status errors, and business‑level exceptions.

Incident Overview

A recent RT jitter incident was initially mitigated by scaling, but the attribution process revealed several investigative steps:

Alert feedback showed increased service RT and goroutine counts.

Grafana revealed a traffic spike with a clear QPS rise.

HTTP/GRPC metrics indicated higher average RT and 99th‑percentile values.

MySQL RT surged, suggesting large or slow queries.

Redis RT also rose, hinting at Redis jitter that could cause timeouts and shift traffic to MySQL.

Routing Logic Analysis

Log inspection revealed redis timeout errors and third‑party service timeouts, confirming the issue was at the service level rather than the underlying infrastructure. CPU and memory metrics showed no bottlenecks, allowing those causes to be ruled out.

Because the service runs multiple pods across different nodes, a single‑pod failure was excluded. Other services in the same cluster remained healthy, eliminating a network‑wide fault. The investigation therefore narrowed to two possibilities: storage‑layer failures or traffic‑path node problems.

Eliminating Storage‑Layer Issues

Using Alibaba Cloud RDS, both MySQL and Redis performance appeared normal with no slow‑query logs, sufficient resources, and stable network bandwidth. A cross‑service check showed another service sharing the same storage layer operated normally during the alert window, allowing storage‑layer faults to be dismissed.

Focusing on the Traffic‑Path Node

The service employs Istio as a service mesh. Initial Istio monitoring seemed fine, but the Istio load reported by the dashboard conflicted with observations from the operations team. After correcting the monitoring data collection, the true Istio load was found to be excessively high, directly correlating with the alert.

The sidecar resources were fixed at 2 CPU + 1 GB RAM, while the pod configuration had been upgraded to 4 CPU + 2 GB RAM. The increased pod resources caused the sidecar pool to be insufficient for the traffic volume, leading to Istio CPU overload and the observed jitter.

Solution: downgrade pod resources to 1 CPU + 2 GB RAM and increase the number of pods (maintaining a 1:1 pod‑to‑sidecar ratio) to expand the sidecar resource pool, thereby preventing similar incidents.

Impact Levels, Possible Causes, and Reference Checklist

CPU side : temporary traffic spikes, code bugs, service scaling, scheduled scripts.

Memory side : temporary traffic spikes, code bugs, service scaling (distinguish RSS vs. cache in k8s).

MySQL/Redis : traffic spikes, slow queries, large batch queries, resource shortages, HA failover.

Traffic‑path nodes : ingress issues (north‑south) or Istio problems (east‑west).

Reference Thinking Process

When an alert arrives, first determine the affected scope, then enumerate possible causes, and finally eliminate candidates based on current conditions. The funnel‑like investigation ends with the root cause.

Quick elimination cases include:

If other services sharing the same storage layer are healthy, storage can be ruled out.

Multiple pods across different ECS instances rule out a single‑node network fault.

If not all traffic entry/exit points are failing, the traffic‑path node can be excluded.

Traffic Path and Storage Layer Overview

North‑South Traffic : Ingress is the critical path; failures can render the entire k8s cluster unavailable.

East‑West Traffic : Envoy proxy handles all internal traffic; proxy issues affect service pods.

MySQL HA Architecture : Multi‑AZ deployment with automatic failover can cause brief service jitter during switchovers.

Redis HA Architecture : Proxy‑based cluster mode with master‑slave replication; automatic or manual failovers and resource changes can also trigger jitter.

Conclusion

Effective alert attribution requires not only solid code‑level debugging skills but also a deep understanding of the overall system architecture, including traffic paths, storage layers, and service‑mesh components. By following a structured funnel‑like analysis and leveraging cross‑service observations, teams can quickly pinpoint root causes and implement preventive measures.