K8s Monitoring Shows Healthy, Users Complain of Lag: 5 eBPF Discoveries That Explain Why

Background: Healthy Dashboards, Unhappy Users

The author's Kubernetes cluster ran Prometheus, Grafana, and Alertmanager with all standard alerts configured. P99 latency, error rates, CPU, and memory all appeared normal. Yet users reported intermittent "lag" — pages would hang briefly, then recover on refresh. No anomalies appeared in any dashboard.

Why eBPF?

eBPF allows running custom programs in kernel space to observe system calls, network packets, and file operations with minimal overhead — no application changes, restarts, or instrumentation required. Tools like Cilium, Falco, and Pixie already use it for networking, security, and observability. The author used it to answer: why do users feel lag when metrics look perfect?

Discovery 1: Micro-Burst Traffic Invisible to Prometheus

Symptom

Users felt "lag" daily between 14:00–15:00. Prometheus showed normal QPS, P99, and error rates.

eBPF Detection

Using

kubectl trace run -e 'kprobe:tcp_sendmsg { @[comm] = count(); }' nginx-pod

on the gateway pod, the author captured per-second packet and byte counts. Every few minutes, a 200–500 ms traffic spike reached 8–10× baseline.

Root Cause

A cron job called an internal API every 5 minutes without rate limiting, causing instantaneous concurrency spikes that queued at the gateway.

Why Prometheus Missed It

Default scrape interval (15–30 s) averages out sub-second bursts — like a slow shutter blurring a fast car.

Fix

Added token-bucket rate limiting to the cron job, smoothing the request curve.

Discovery 2: TCP Zero Window Causing Frontend "Fake Hangs"

Symptom

Frontend reported occasional page hangs (5–30 s) with requests stuck in pending state. Backend logs showed 200 OK, no errors, normal QPS and P99.

eBPF Detection

On the backend pod's node, the author traced tcp_send_probe0 (kernel function for zero-window probes) via bpftrace. Abnormally high probe rates pointed to one backend pod. Further tracing showed its TCP receive buffer filled rapidly during external API calls, closing the window and stalling data in the kernel.

Root Cause

The backend aggregation service fetched tens of MB from an external API. With only 512 MiB memory limit, the application couldn't consume data fast enough. The TCP receive buffer saturated, kernel advertised window=0, pausing the external sender. The request entered a "fake hang": frontend waiting on backend, backend waiting on external API, external API waiting for window reopen.

Why Monitoring Missed It

APM measures request entry-to-exit time. While blocked in kernel buffer wait, the request hasn't exited, so that latency is excluded from P99/average.

Fix

Increased memory limit from 512 MiB to 2 GiB.

Raised OS TCP receive buffer: net.ipv4.tcp_rmem = 4096 87380 16777216.

Refactored to finer-grained streaming consumption instead of buffering entire response.

Result: worst-case page load dropped from 30 s to under 2 s consistently.

Discovery 3: Rogue fsync Storm Nearly Triggered MySQL Failover

Symptom

MySQL primary stalled for tens of seconds. CPU, memory, disk I/O all normal. SHOW PROCESSLIST showed many queries in "Waiting for table flush".

eBPF Detection

Ran biosnoop-bpfcc (bcc-tools) to trace block I/O. Every few minutes, a non-MySQL PID issued massive fsync calls, each taking 200–500 ms. The PID belonged to a Java application on the same node, sharing the same disk.

Root Cause

The Java app used Logback with immediateFlush=true, forcing fsync on every log write (thousands per minute). Frequent fsync flushes the filesystem journal, serializing other fsync callers — including MySQL — on the shared journal. This journal-layer contention amplified latency under load.

Why Monitoring Missed It

Disk I/O monitors track throughput and IOPS. fsync calls are small, low-count, high-latency operations invisible to throughput metrics.

Fix

Disabled immediateFlush, switched to async flushing.

Separated MySQL and log-heavy workloads onto different nodes.

Provisioned dedicated SSD for MySQL.

Risk: prolonged primary stall could have triggered automatic failover with potential data inconsistency.

Discovery 4: TCP Retransmissions Concentrated on One Pod — Not a Network Issue

Symptom

Service error rate spiked from 0.1% to 2–3% for minutes, showing "connection reset by peer". Network tools (ping, mtr) and peer service metrics were normal.

eBPF Detection

Traced tcp_retransmit_skb via bpftrace:

bpftrace -e 'kprobe:tcp_retransmit_skb { @[args->sk->__sk_common.skc_daddr] = count(); }'

. Retransmissions clustered on a specific backend pod IP. That pod's conntrack table was full.

Root Cause

The pod ran a long-connection gateway; each connection consumes a conntrack entry. Default conntrack_max=65536 was exhausted. New connections were dropped, manifesting as TCP retransmissions.

Why Monitoring Missed It

Overall TCP retransmit rate stayed normal — the single problematic pod's signal was averaged out.

Fix

Increased conntrack_max.

Set hostNetwork: true on the pod to bypass conntrack.

Evaluated eBPF-based connection tracking as conntrack replacement.

Discovery 5: Container Startup Traversing 100,000 Files

Symptom

One service's rolling updates took 2–3 minutes per pod vs. 20–30 s for others. Image size only 200 MB; readiness probe initialDelaySeconds=10.

eBPF Detection

Traced sys_enter_openat via bpftrace during startup:

bpftrace -e 't:syscalls:sys_enter_openat { @[comm] = count(); }' -p $(pgrep -n myapp)

. The process opened 100,000+ files at startup.

Root Cause

Image contained a node_modules directory with 100k+ small files. Startup script ran chown -R app:app /app, recursively changing ownership on every file at each container start.

Fix

Fixed file ownership at image build time, not runtime.

Used multi-stage build to isolate node_modules in its own layer.

Considered fsGroup instead of chown.

eBPF vs. Prometheus: Complementary, Not Replacement

The author summarizes the distinction across five dimensions:

Observation Granularity : Prometheus provides aggregated metrics (P99, QPS, error rate); eBPF provides per-request, per-system-call visibility.

Time Precision : Prometheus operates at second- or minute-level; eBPF at millisecond- or microsecond-level.

Coverage : Prometheus covers application layer; eBPF covers kernel + application layer.

Overhead : Prometheus has low overhead; eBPF has very low overhead (depends on BPF program complexity).

Ease of Use : Prometheus is high; eBPF is low (requires kernel internals knowledge).

eBPF fills Prometheus's blind spots but has costs: steep learning curve (kernel, syscalls, network stack), immature tooling vs. Prometheus ecosystem, production risk (kernel crashes from buggy BPF programs despite verifier), and massive data volume (1000× Prometheus) raising storage, analysis, and alerting challenges.

Advice for Operations Engineers

Don't become an eBPF expert, but know what problems it solves. Consider eBPF when: metrics are healthy but users report issues; intermittent latency spikes or errors occur; kernel-layer behavior (syscalls, packets, file ops) must be observed; traditional tools hit blind spots.

Start with network troubleshooting — lowest barrier. Cilium already wraps eBPF; use cilium monitor, hubble observe.

Closing Reflection

Traditional observability (metrics, logs, traces) sees only the "tip of the iceberg." eBPF reveals the submerged kernel-layer issues — often the hardest to debug. However, eBPF sees runtime behavior, not configuration risks (e.g., excessive RBAC, anonymous API server access, hostPath mounts). The author plans a follow-up security audit, which uncovered 23 high-severity vulnerabilities, 3 still haunting.