Master Linux Kernel Tracing: From Static Tracepoints to Dynamic Kprobes

1. Linux Tracing Overview

Tracing (or "tracing" in English) is a large and rapidly evolving field in advanced Linux usage, with new eBPF‑based tools emerging in recent years. Tracing techniques are divided into event sources and tools: event sources generate raw data, while tools process and present it to users.

Common tools include flame graphs (built on perf or dtrace), ftrace, trace‑cmd, and the newer eBPF ecosystem with BCC and bpftrace.

Event sources consist of hardware events, software events, tracepoints, kprobes, uprobes, and USDT. Hardware and software events can be listed with perf list hw and perf list sw respectively.

2. Kernel Source Tracing

The author primarily uses IDE navigation and grep for most cases, but demonstrates how to perform static and dynamic tracing with ftrace and perf.

2.1 Static Tracing

Static tracepoints are listed under /sys/kernel/debug/tracing/events/. For example, to explore the sched:sched_switch tracepoint:

# ls /sys/kernel/debug/tracing/events/

# ls /sys/kernel/debug/tracing/events/sched/

The tracepoint can be enabled by writing 1 to its enable file and observed via cat /sys/kernel/debug/tracing/trace_pipe:

# echo 1 > /sys/kernel/debug/tracing/events/sched/sched_switch/enable
# cat /sys/kernel/debug/tracing/trace_pipe

After tracing, disable the point with echo 0 …/enable. The same tracepoint can also be recorded with perf record -e 'sched:sched_switch' -a sleep 3 and later inspected with perf script. Adding -g records the call stack.

2.2 Dynamic Tracing (kprobes)

Dynamic tracing uses kprobes, which insert a breakpoint instruction at the target address. A probe is created by writing to kprobe_events:

# cd /sys/kernel/debug/tracing
# echo 'p:myprobe schedule' >> kprobe_events

The new probe appears under events/kprobes/myprobe. Enabling it and reading trace_pipe shows runtime output, but perf record -e kprobes:myprobe -a -g sleep 1 captures the call stack.

# perf record -e kprobes:myprobe -a -g sleep 1
# perf script

3. Tracing Implementation Details

3.1 Static Tracepoints

Static tracepoints are defined by macros in the kernel source. DEFINE_TRACE(name) creates a struct tracepoint named __tracepoint_##name. The structure contains the name, a static key indicating enable/disable, registration functions, and a list of hooked functions.

struct tracepoint {
    const char *name;
    struct static_key key;
    int (*regfunc)(void);
    void (*unregfunc)(void);
    struct tracepoint_func __rcu *funcs;
};

DECLARE_TRACE(name, proto, args)

expands to inline functions such as trace_name() that check the static key and, if enabled, invoke __DO_TRACE to iterate over all registered callbacks.

#define __DO_TRACE(tp, proto, args, cond, rcuidle) \
    do { \
        it_func_ptr = rcu_dereference_raw((tp)->funcs); \
        if (it_func_ptr) { \
            do { \
                it_func = (it_func_ptr)->func; \
                __data = (it_func_ptr)->data; \
                ((void (*)(proto))(it_func))(args); \
            } while ((++it_func_ptr)->func); \
        } \
    } while (0)

If a tracepoint is disabled, the inline function returns immediately, incurring virtually no overhead.

3.2 Dynamic Tracing with kprobes

kprobes replace the target instruction with a breakpoint (e.g., INT3 on x86). The registration flow is:

Locate the address of the target symbol ( kprobe_addr).

Save the original instruction ( prepare_kprobe).

Replace it with the breakpoint ( arm_kprobe).

int register_kprobe(struct kprobe *p) {
    addr = kprobe_addr(p);
    p->addr = addr;
    prepare_kprobe(p);
    arm_kprobe(p);
    return 0;
}

On x86, arm_kprobe uses text_poke to write the breakpoint instruction:

void arch_arm_kprobe(struct kprobe *p) {
    text_poke(p->addr, (unsigned char[]){BREAKPOINT_INSTRUCTION}, 1);
}

When the kernel executes the breakpoint, the kprobe_int3_handler is invoked, which calls the registered pre_handler, optionally sets up single‑step to execute the original instruction, and then resumes normal execution.

int kprobe_int3_handler(struct pt_regs *regs) {
    p = get_kprobe(addr);
    if (!p->pre_handler || !p->pre_handler(p, regs))
        setup_singlestep(p, regs, kcb, 0);
    return 0;
}

4. Summary

The article covered three parts: an overview of Linux tracing technologies (perf, ftrace, eBPF, etc.), practical demonstrations of static and dynamic kernel tracing, and a deep dive into the kernel‑level implementation of tracepoints and kprobes. Understanding these mechanisms helps developers observe system behavior, diagnose performance bottlenecks, and perform precise kernel debugging.