Essential eBPF Tracing Performance Tuning: What Every Developer Must Know

Hook mechanisms

kprobe

kprobe inserts a breakpoint instruction (e.g., int3 on x86) at the start of the target function. When the CPU executes this instruction, an exception is raised, the kernel captures it, and calls the registered probe handler where the eBPF program runs. Linux also provides an optimized variant kprobe‑optimized that jumps directly to the probe handler, avoiding the breakpoint overhead.

tp & raw_tp

tracepoint

is a static tracing mechanism defined by kernel developers. Because tracepoint locations are fixed in the kernel source, they have lower overhead and higher stability than kprobe. In eBPF, the tp variant places the eBPF program in the traditional tracepoint callback, while raw_tp generates an additional bpf_trace_runxx function, allowing earlier access to the original tracepoint parameters.

fentry

fentry (trampoline) replaces a NOP instruction in the target kernel function and jumps directly to the eBPF program, eliminating the context‑switch overhead. Compared with tracepoint, fentry reaches the eBPF program with fewer intermediate instructions.

Performance comparison

A benchmark was executed on QEMU with Linux kernel 6.13 on x86. The latency results confirm the analysis: fentry shows the lowest latency, followed by raw_tp, tp, and kprobe.

Kernel version support

kprobe : x86 4.1, arm64 4.1

tp : x86 4.7, arm64 4.7

raw_tp : x86 4.17, arm64 4.17

fentry : x86 5.5, arm64 6.0

Conclusion

None of the hook methods is intrinsically superior; the choice depends on the production scenario. Because fentry reaches the eBPF program with the shortest call chain, it offers the best theoretical performance.

Reference:

https://github.com/torvalds/linux