How to Dynamically Trace Kernel Functions with eBPF Using Last Branch Record

Last Branch Record (LBR) is a CPU hardware feature that records branch‑jump information during program execution. Intel, AMD and, in progress, ARM CPUs support LBR.

On an Intel PC LBR support can be verified with:

$ dmesg | grep -i lbr
[    0.101727] Performance Events: XSAVE Architectural LBR, PEBS fmt4+-baseline,  AnyThread deprecated, Alderlake Hybrid events, 32-deep LBR, full-width counters, Intel PMU driver.

Kernel documentation and the Intel® 64 and IA‑32 Architectures Software Developer Manuals, together with LWN articles “An introduction to last branch records” and “Advanced usage of last branch records”, provide further details.

bpflbr architecture

The commit bpf: Introduce helper bpf_get_branch_snapshot (Linux 5.16) added the bpf_get_branch_snapshot helper, which retrieves the current CPU’s LBR records. The bpflbr tool implements this helper in an eBPF program and provides a userspace component to decode the records. It consists of two parts: bpflbr BPF program that collects LBR records. bpflbr userspace program that parses the records.

Because LBR records branch jumps, the BPF program must call bpf_get_branch_snapshot at its very beginning to capture the LBR snapshot before the program itself generates additional branch entries.

struct event {
    struct perf_branch_entry lbr[MAX_LBR_ENTRIES];
    // ...
} __attribute__((packed));

static __always_inline int emit_lbr_event(void *ctx) {
    struct event *event;
    // ...
    if (!cfg->suppress_lbr)
        event->nr_bytes = bpf_get_branch_snapshot(event->lbr, sizeof(event->lbr), 0);
    bpf_ringbuf_output(&events, event, sizeof(*event), 0);
    return BPF_OK;
}

Userspace decoding

If the record refers to a kernel function, resolve the symbol name via /proc/kallsyms and obtain line information from the vmlinux debug symbols.

If the record refers to a BPF program, pre‑parse all BPF program metadata (function names, line info) and map the LBR entries to those symbols.

Output the branch‑jump information as a call stack.

Feature list

Dynamic tracing of BPF program internals using fexit.

Dynamic tracing of kernel function branch records before BPF execution using fentry.

Output of function call stacks via --output-stack.

Detailed output of function parameters and return values.

Disassembly of kernel functions and BPF programs using the Capstone engine.

Command‑line options

$ ./bpflbr -h
Usage of bpflbr:
  -d, --disasm               disasm bpf prog or kernel function
  -B, --disasm-bytes uint    disasm bytes of kernel function, 0 to guess automatically
      --disasm-intel-syntax  use Intel asm syntax for disasm, ATT syntax by default
      --filter-pid uint32    filter pid for tracing
  -k, --kfunc strings        filter kernel functions by shell wildcards
      --kfunc-all-kmods      filter functions in all kernel modules
      --limit-events uint    limited number events to output, 0 to output all events
  -m, --mode string          mode of lbr tracing, exit or entry (default "exit")
  -o, --output string        output file for the result, default is stdout
      --output-stack         output function call stack
  -p, --prog strings         bpf prog info for bpflbr in format PROG[,PROG,..]
      --suppress-lbr         suppress LBR perf event
  -v, --verbose              output verbose log

Demonstrations

Future work

Planned extensions include integrating dynamic function‑parameter filtering from the “eBPF Talk: Dynamic Filtering of Function Parameters” talk, adding pcap‑filter(7) syntax for packet filtering, and referencing the bice project to selectively output fields of struct/union pointer arguments.

Summary

bpflbr

leverages the CPU’s LBR feature to trace kernel functions and BPF programs, providing call‑stack output, detailed argument/return information, and disassembly of both kernel and BPF code.

References

Intel® 64 and IA‑32 Architectures Software Developer Manuals – https://www.intel.com/content/www/us/en/developer/articles/technical/intel-sdm.html

An introduction to last branch records – https://lwn.net/Articles/680985/

Advanced usage of last branch records – https://lwn.net/Articles/680996/

bpf: Introduce helper bpf_get_branch_snapshot – https://github.com/torvalds/linux/commit/856c02dbce4f

Capstone engine – https://github.com/capstone-engine/capstone

bice – https://github.com/leonhwangprojects/bice

bpflbr source repository – https://github.com/Asphaltt/bpflbr