retsnoop: Kernel Error Debugging Tool that Traces All Functions and Shows Stack on Failure

retsnoop is an eBPF‑based tracing utility that lets users define wildcard patterns to hook all matching kernel functions and automatically displays stack information when a function returns an error.

Quick Start

Typical usage invokes the binary with a pattern, for example:

$ sudo ./retsnoop -e '*sys_bpf' -a ':kernel/bpf/*.c'

This command traces all functions matching *sys_bpf in the kernel BPF source tree and prints the deepest call stack for each error return, including timestamps, PID/TID, and source‑code locations.

Supported Modes

Stack trace mode (default)

Shows the deepest call stack that satisfies a user‑defined condition (e.g., a non‑zero return). Each frame is printed with file, line, offset, latency, and return value.

Function‑call trace mode ( -T )

Provides a detailed control‑flow trace for a set of functions, displaying each call, its result, and duration.

LBR mode

Uses Intel CPU Last Branch Record hardware to rewind the execution path, revealing inlined functions and pinpointing the exact C statement that caused a problem. Requires Linux kernel 5.16 or newer.

Additional filters include PID filtering ( -p/--pid and -P/--no-pid) and duration filtering ( -L/--longer), which limit output to specific processes or to stacks exceeding a given latency.

Source‑Code Overview

retsnoop relies on advanced eBPF features such as fentry, kprobe, kprobe_multi, ring buffers, global variables, and LBR support, thus requiring Linux 5.16+ for full functionality. Main source files: calib_feat.bpf.c: Detects which high‑level eBPF features the current kernel supports (e.g., ring buffer, multi‑probe, LBR). mass_attacher.c: Implements entry hooks for all target functions, handling handle_func_entry and handle_func_exit. retsnoop.bpf.c: Contains core logic that records timestamps, sequence IDs, decides whether a return value is abnormal, and emits the stack trace.

Implementation Steps

Users supply wildcard patterns (e.g., sys_bpf ) that match target kernel functions.

Each matched function is hooked at entry and exit; entry records start time and a unique seq_id, exit checks the return value (non‑zero, non‑NULL, etc.).

If the return is abnormal, the tool captures the current stack depth; when the depth reaches zero (function return), the accumulated trace is emitted.

Use Cases

Per‑CPU map value size limit (32 KB)

A tool attempted to allocate a per‑CPU map larger than 32 KB; the kernel returned NULL with “Cannot allocate memory”. retsnoop revealed that the failure originated from the kernel constant PCPU_MIN_UNIT_SIZE (32 KB), confirming the size limitation.

AArch64 atomic instruction not supported

On a 5.15 AArch64 kernel, loading an eBPF program produced error “failed to load: -524”. Traditional verifier logs were absent. Using retsnoop to trace bpf_prog_load showed an ENOTSUPPORT error triggered by the use of the sync_fetch_and_and atomic instruction, which the kernel did not yet implement for that architecture.

Portability Note

Many retsnoop features depend on newer eBPF capabilities; the tool works best on kernels 5.16 and above. Users may backport a minimal version that only uses kprobe and perf buffers on older kernels such as 4.19.

Reference

https://github.com/anakryiko/retsnoop