Mastering Linux Kernel Debugging: Tools, Filesystems, and Tracing Techniques

Linux kernel debugging differs from user‑space debugging and requires specialized tools and mechanisms to exchange data between kernel and user space.

1. Kernel Debugging Overview

The kernel provides several pseudo filesystems— procfs, sysfs, debugfs, and relayfs —to expose kernel data and parameters to user space. Each serves distinct scenarios: procfs is the oldest, offering read‑only system information; sysfs is tightly coupled with the kobject framework for device driver attributes; debugfs is a flexible debugging interface; relayfs efficiently forwards large data streams via channels.

2. Pseudo Filesystem Details

procfs

: historic interface for kernel‑user data exchange, exposing CPU, memory, and process info under /proc. Most entries are read‑only. sysfs: memory‑based filesystem built on ramfs, exposing kernel objects and attributes via the kobject model. Users can read or modify module parameters by mounting sysfs and accessing files under /sys. debugfs: created by Greg Kroah‑Hartman (2.6.11), provides a small, easy‑to‑use virtual filesystem for debugging output. It avoids the performance penalties of printk for large data. relayfs: designed for high‑throughput kernel‑to‑user data transfer. Data is written to per‑CPU buffers (channels) and read via standard open, mmap, poll, and close operations.

Typical mount commands:

mkdir -p /sysfs
mount -t sysfs sysfs /sysfs
mkdir -p /mnt/relay
mount -t relayfs relayfs /mnt/relay

3. Printk

printk

is the kernel’s analogue of printf. While simple, it always emits output and can clutter logs, making selective tracing preferable.

4. Tracing with ftrace & trace‑cmd

ftrace

is the most powerful built‑in tracing framework, offering static (compiled‑in) and dynamic (via mcount) probe points. It supports multiple tracers (function, irq, preempt, etc.) and a plugin architecture.

The front‑end trace‑cmd simplifies data collection and reporting:

# Collect tracing data
sudo trace‑cmd record -e sched_switch
# Generate a report
sudo trace‑cmd report

5. Dynamic Probing with kprobe & systemtap

kprobe

allows insertion of probes at arbitrary kernel instructions, with three probe types: kprobe (any instruction), jprobe (function entry), and kretprobe (function return). It underpins higher‑level tools like perf and systemtap. systemtap provides a scripting language (similar to DTrace) to define probes and actions dynamically, leveraging the kprobe API. Example usage:

stap -e 'probe kernel.function("do_fork") { printf("fork called
") }'

6. KGTP – Lightweight Remote Debugging

KGTP

offers a remote GDB interface without requiring kernel patches. By loading the kgtp module, developers can set GDB tracepoints and perform offline debugging, supporting x86, ARM, and MIPS architectures.

7. Performance Analysis with perf

perf

(Performance Events) evolved from simple PMU counters to a full‑featured event framework handling hardware, software, and tracepoint events. It can measure IPC, per‑thread statistics, function‑level sampling, and replace tools like strace for system‑call tracing.

# Record CPU cycles and instructions
perf record -e cycles,instructions -a
# Generate a report
perf report

Perf also supports benchmarking schedulers and generating flame graphs.

8. Additional Tracing Solutions

LTTng : Open‑source Linux tracing suite with kernel and user‑space components.

eBPF : In‑kernel virtual machine enabling JIT‑compiled programs for tracing, soon to unify ftrace and perf_events.

Ktap : Lua‑based tracer for embedded devices (now superseded by eBPF).

dtrace4linux : Experimental port of Solaris DTrace to Linux.

Oracle Linux DTrace : Oracle’s effort to bring DTrace to Linux, with providers like syscall, profile, and USDT.

sysdig : tcpdump‑style tracer using Lua scripts, currently limited to system‑call events.

These tools collectively form a robust ecosystem for kernel developers to diagnose, profile, and optimize Linux kernels and drivers.