How eBPF Powers Next‑Gen Observability and Root‑Cause Analysis in Kubernetes

The presentation, based on a KubeCon China 2023 session, begins by outlining three key observability challenges in Kubernetes: (1) the prevalence of network‑related incidents (over 56% of tickets), (2) the need to collect data from every stack layer (application, container, network, kernel), and (3) the difficulty of correlating siloed metrics.

eBPF‑Based Data Collection

eBPF, a virtual machine running inside the Linux kernel, allows custom logic to be injected without recompiling the kernel or restarting services. By writing eBPF programs, compiling them to bytecode, and attaching them via bpf system calls to various hook points (system calls, kernel functions, or user‑space entry/exit), developers can capture rich telemetry with three main properties:

Non‑intrusive : dynamic attachment without process restarts.

High‑performance : JIT‑compiled bytecode runs at near‑native speed.

Secure : sandboxed execution with verifier checks.

Typical eBPF attachment points include netif_receive_skb, dev_queue_xmit for network packets, and syscall interfaces such as read, write, sendto, recvfrom for I/O monitoring.

Architecture Awareness

By instrumenting kernel networking functions, eBPF can automatically discover service topology, traffic flows, and network quality metrics (packet counts, sizes, drops, retransmissions). This creates a unified view of the cluster, highlighting abnormal nodes and links.

Application‑Level Performance Monitoring

Traditional APM probes embed at the RPC library level, which ties them to specific languages and frameworks. eBPF can instead attach at lower layers (syscalls, IP stack, NIC driver), capturing request/response payloads and timing without language coupling. By parsing protocols (e.g., HTTP headers) eBPF extracts method, path, status code, and computes request latency.

Multi‑Dimensional Data Correlation

Because eBPF can capture process IDs, PID namespaces, container IDs, and network 5‑tuple information, it provides a common key to join metrics from containers, Kubernetes resources, and application traces. This eliminates data islands and enables unified dashboards that correlate CPU/memory, pod identifiers, service names, and trace IDs.

Fault Localization Practice

The speaker demonstrates a manual troubleshooting workflow in ARMS K8s monitoring: start from an alarm on a gateway’s response‑time spike, traverse the service topology to pinpoint the downstream service (product service) showing similar anomalies, and finally inspect network metrics (retransmissions, RTT) on the problematic link.

Automating this process involves three steps: dimension attribution (identifying which metric dimensions cause the anomaly), anomaly bounding (isolating the abnormal values), and Fault Tree Analysis (FTA) to classify root causes.

Root‑Cause Analysis Model

Dimension attribution drills down from aggregated metrics to specific dimension values (e.g., service, region, host). Combining dimensions reveals intersecting anomalies (e.g., a particular host‑service pair). Horizontal attribution explores service‑to‑service dependencies, while vertical attribution examines resource‑level dependencies. The final FTA step uses a decision‑tree‑like structure to map observed conditions (e.g., network latency, CPU usage) to concrete fault categories.

Product Integration

The complete workflow is packaged into the Insights product, which provides real‑time anomaly detection, an event list, and detailed root‑cause reports that include phenomenon description, key metrics, correlation graphs, and actionable recommendations. Insights supports multiple monitoring scenarios, including application and Kubernetes monitoring.

In summary, the talk covered the three observability challenges in Kubernetes, presented an eBPF‑driven data‑collection architecture that spans all layers, and shared a practical, automated root‑cause analysis pipeline built on dimension attribution, anomaly bounding, and fault‑tree analysis.