How Tracepoint and kprobe Enable Precise Linux Kernel Source Tracing

1. Linux Tracing Overview

Linux tracing encompasses a large set of event sources and tools. Event sources such as hardware events, software events, tracepoints, kprobes, uprobe, and USDT provide raw data, while tools like flame graphs, perf, ftrace, trace‑cmd, eBPF, BCC, and bpftrace process and display this data.

Understanding the event sources is essential because all tracing tools rely on them.

2. Kernel Source Tracing Methods

The author typically uses two approaches: (1) navigating the source with an IDE or grep, and (2) employing Linux tracing technologies such as strace, tracepoints, and kprobes.

2.1 Static Tracing with Tracepoints

Static tracepoints are predefined hooks inserted in the kernel source. They are disabled by default and can be enabled at runtime. Example: the sched:sched_switch tracepoint is placed in __schedule and prints process information when enabled.

# ls /sys/kernel/debug/tracing/events/
# ls /sys/kernel/debug/tracing/events/sched/
# echo 1 > /sys/kernel/debug/tracing/events/sched/sched_switch/enable
# cat /sys/kernel/debug/tracing/trace_pipe

Using perf to record the same event and capture call stacks:

# perf record -e 'sched:sched_switch' -a -g sleep 3
# perf script

Static tracepoints have low overhead when disabled because the inline function returns immediately.

2.2 Dynamic Tracing with kprobes

kprobes allow inserting probes into arbitrary kernel functions at runtime. They require the kernel to be compiled with CONFIG_KPROBE_EVENT. A probe is created by writing to kprobe_events:

# cd /sys/kernel/debug/tracing
# echo 'p:myprobe schedule' >> kprobe_events

The new probe appears under events/kprobes/myprobe. Enabling the probe and reading trace_pipe shows dynamic output, while perf can record call stacks:

# perf record -e kprobes:myprobe -a -g sleep 1
# perf script

3. Implementation Details

3.1 Static Tracepoint Mechanics

Three macros implement static tracepoints:

DEFINE_TRACE : defines a struct tracepoint named __tracepoint_#name.

DECLARE_TRACE : declares and implements the associated trace_xxx and register_trace_xxx functions.

DO_TRACE : executes all registered hook functions when the tracepoint is enabled.

The struct tracepoint contains the name, a static key indicating enable state, registration callbacks, and a list of hook functions. When enabled, the macro expands to an inline function that checks the static key and calls __DO_TRACE, which iterates over tp->funcs and invokes each probe.

3.2 Dynamic kprobe Mechanics

kprobes replace the target instruction with a BREAKPOINT_INSTRUCTION and store the original instruction. The core steps in register_kprobe are:

Locate the target address ( kprobe_addr).

Save the original instruction ( prepare_kprobe).

Replace it with the breakpoint ( arm_kprobe), which on x86 calls text_poke.

When the kernel executes the breakpoint, an INT3 exception triggers kprobe_int3_handler, which invokes the pre‑handler, optionally records data, and then restores the original instruction before resuming execution.

4. Summary

The article covered three parts: an overview of Linux tracing technologies, practical demonstrations of static and dynamic kernel tracing using ftrace and perf, and a deep dive into the implementation of tracepoints and kprobes. Both mechanisms are conceptually simple—tracepoints insert static hooks, while kprobes dynamically replace instructions with breakpoints to capture execution.