Understanding and Using Kprobes for Dynamic Kernel Debugging

A developer once faced intermittent kernel stalls and, after traditional debugging proved cumbersome, discovered Kprobes—a dynamic debugging mechanism that allows probing kernel functions without source changes or reboot.

1. Overview of Kprobes

Kprobes lets developers insert probe points at function entry or specific offsets, capturing arguments, return values, and execution time, greatly simplifying performance analysis and bug isolation.

2. How Kprobes Works

When a probe is registered, the target instruction is replaced with a breakpoint (int3 on x86). The CPU traps, saves registers, and invokes the probe’s pre‑handler, executes the original instruction in single‑step mode, then calls the post‑handler before resuming normal execution.

2.1 Key Data Structure

<code>struct kprobe {
    kprobe_opcode_t *addr;            // address of probe point
    const char *symbol_name;         // function name
    unsigned int offset;             // offset inside function (0 = entry)
    kprobe_pre_handler_t pre_handler;
    kprobe_post_handler_t post_handler;
    kprobe_fault_handler_t fault_handler;
    kprobe_opcode_t opcode;          // original instruction
    struct arch_specific_insn ainsn;
    u32 flags;                       // status flags
};</code>

2.2 Registration and Unregistration Flow

To probe a function, create a struct kprobe instance, set .symbol_name and handler callbacks, then call register_kprobe(&kp) . On module unload, call unregister_kprobe(&kp) .

<code>#include <linux/module.h>
#include <linux/kprobes.h>

static int handler_pre(struct kprobe *p, struct pt_regs *regs) {
    pr_info("<%s> pre_handler: p->addr = %p, ip = %lx, flags = %lx\n",
            p->symbol_name, p->addr, regs->ip, regs->flags);
    return 0;
}

static void handler_post(struct kprobe *p, struct pt_regs *regs, unsigned long flags) {
    pr_info("<%s> post_handler: p->addr = %p, flags = %lx\n",
            p->symbol_name, p->addr, flags);
}

static int handler_fault(struct kprobe *p, struct pt_regs *regs, int trapnr) {
    pr_info("fault_handler: p->addr = %p, trap #%d\n", p->addr, trapnr);
    return 0;
}

static struct kprobe kp = {
    .symbol_name = "do_fork",
    .pre_handler = handler_pre,
    .post_handler = handler_post,
    .fault_handler = handler_fault,
};

static int __init kprobe_init(void) {
    int ret = register_kprobe(&kp);
    if (ret < 0) {
        pr_err("register_kprobe failed, returned %d\n", ret);
        return ret;
    }
    pr_info("Planted kprobe at %p\n", kp.addr);
    return 0;
}

static void __exit kprobe_exit(void) {
    unregister_kprobe(&kp);
    pr_info("kprobe at %p unregistered\n", kp.addr);
}

module_init(kprobe_init);
module_exit(kprobe_exit);
MODULE_LICENSE("GPL");</code>

3. Implementation Details

The probe replaces the target instruction with a breakpoint, saves the original opcode, and uses the kernel’s notifier chain to invoke the registered handlers. Kretprobes work similarly but replace the return address with a trampoline to capture return values.

4. Limitations

Multiple probes can share an address, but jprobes cannot coexist on the same address. Inline functions may not be fully probed, and handlers must avoid sleeping or acquiring semaphores because they run in interrupt context.

5. Enabling Kprobes in the Kernel

Set CONFIG_KPROBES (and related options such as CONFIG_KPROBES_ON_FTRACE ) in the kernel configuration, compile, and install the kernel. Ensure module support ( CONFIG_MODULES ) and symbol lookup ( CONFIG_KALLSYMS ) are enabled.

6. Practical Usage Examples

6.1 Simple Kprobe Module for do_sys_open

<code>#include <linux/module.h>
#include <linux/kprobes.h>
#include <linux/sched.h>

static int count = 0;

static int handler_pre(struct kprobe *p, struct pt_regs *regs) {
    char *filename = (char *)regs->di;
    int flags = (int)regs->si;
    printk(KERN_INFO "do_sys_open called with filename=%s, flags=%x\n", filename, flags);
    count++;
    return 0;
}

static struct kprobe kp = {
    .symbol_name = "do_sys_open",
    .pre_handler = handler_pre,
};

static int __init mymodule_init(void) {
    int ret = register_kprobe(&kp);
    if (ret < 0) {
        printk(KERN_INFO "register_kprobe failed\n");
        return ret;
    }
    printk(KERN_INFO "kprobe registered\n");
    return 0;
}

static void __exit mymodule_exit(void) {
    unregister_kprobe(&kp);
    printk(KERN_INFO "kprobe unregistered\n");
    printk(KERN_INFO "do_sys_open called %d times\n", count);
}

module_init(mymodule_init);
module_exit(mymodule_exit);
MODULE_LICENSE("GPL");</code>

6.2 Using Kprobes via ftrace

Enable the following kernel options:

<code>CONFIG_KPROBES=y
CONFIG_OPTPROBES=y
CONFIG_KPROBES_ON_FTRACE=y
CONFIG_UPROBES=y
CONFIG_KRETPROBES=y
CONFIG_KPROBE_EVENT=y</code>

Register a probe by writing to /sys/kernel/debug/tracing/kprobe_events :

<code>echo 'p:myprobe do_sys_open dfd=%ax filename=%dx flags=%cx mode=+4($stack)' > /sys/kernel/debug/tracing/kprobe_events
echo 'r:myretprobe do_sys_open ret=$retval' >> /sys/kernel/debug/tracing/kprobe_events</code>

Enable the event:

<code>echo 1 > /sys/kernel/debug/tracing/events/kprobes/myprobe/enable
echo 1 > /sys/kernel/debug/tracing/events/kprobes/myretprobe/enable</code>

Results appear in /sys/kernel/debug/tracing/trace and can be filtered via the filter file.

7. Troubleshooting

Common issues include failure to register a probe (missing symbol, protected code), or handlers crashing due to illegal memory access. Verify symbols with /proc/kallsyms , avoid sleeping in handlers, and use work queues for complex processing.