How Kindling Leverages eBPF to Reach 1‑5‑10 Observability Targets

Background

Production observability often targets the "1‑5‑10" goal: detect an incident within 1 minute, respond within 5 minutes, and recover within 10 minutes. Existing tracing, logging and metrics stacks (e.g., SkyWalking, Pinpoint, OpenTelemetry, Prometheus, Zabbix, ELK) still require deep expertise and give unpredictable resolution times.

Current Challenges

Proliferation of metrics makes it unclear which ones to monitor at any moment.

Frequent abnormal metric spikes extend investigation cycles.

Resolution time varies widely, making the 10‑minute recovery target unrealistic.

Kindling Overview

Kindling is an open‑source cloud‑native observability project that combines tracing, logging and metrics through eBPF‑based trace‑profiling. It records the full execution path of a request—from kernel to application—and maps resource consumption (CPU, memory, network, storage) to trace spans, providing a standardized, minute‑level root‑cause view.

Trace‑Profiling Technique

The system captures a request as a series of spans. Each span is enriched with resource‑level metrics, allowing users to quickly pinpoint whether CPU, network, storage or application layers are the bottleneck. By correlating these spans with logs and metrics, Kindling presents a unified view that guides analysts to the most relevant indicators.

Root‑Cause Diagnosis Workflow

Locate the relevant trace for the incident.

Inspect span‑level resource consumption (CPU, network, storage, memory).

Drill down to specific metrics associated with the hotspot span.

Correlate with logs/metrics to confirm the underlying cause.

Use Cases

Intermittent CPU Spikes in Production

Traditional debugging requires manual SSH, process inspection and jstack, which is time‑consuming and often unreproducible. Kindling visualizes CPU consumption per span, revealing that serialization libraries (e.g., fastjson) dominate CPU time, enabling developers to target the exact code path.

Service‑Name Remote Calls in Kubernetes

When a client experiences high latency calling a service by name but the server reports normal execution time, conventional analysis suspects network issues. Kindling’s trace separates client‑side latency from server processing, confirming that the problem lies in the client‑side network path.

Cloud‑Native Storage Performance Problems

Standard metrics rarely surface storage latency. By tracing I/O spans, Kindling highlights prolonged storage‑related spans, exposing slow disk reads/writes or problematic remote storage (e.g., GlusterFS) that would otherwise be missed.

TCP Window Misconfiguration Causing Large‑Packet Delays

Large RPC responses (1‑3 MB) experience near‑second latency despite high bandwidth. Kindling traces show the latency is concentrated in the TCP window handling. Adjusting the TCP window size reduces response time by roughly 50%.

Implementation Details

Kindling uses eBPF programs attached to kernel tracepoints and user‑space probes to collect call stacks, resource counters and network events. The collected data is aggregated into spans with timestamps and resource usage. The backend can export data to OpenTelemetry, Prometheus or ELK for further analysis.

Deployment Example

# Clone the repository
git clone https://github.com/KindlingProject/kindling.git
cd kindling

# Build the eBPF agents
make build

# Run the agent with a configuration file
./kindling-agent --config config.yaml

# Access traces via the Kindling UI or export them to a collector

Benefits

Unified view of tracing, logging and metrics.

Minute‑level root‑cause identification reduces investigation time.

Helps teams approach the 1‑5‑10 observability target.