Mastering Linux Kernel Debugging: Tools, Configurations, and Practical Tips

Why Kernel Debugging Is Hard

Kernel development is tougher than user‑space because bugs often crash the whole system, making it difficult to capture the state at the moment of failure; deep understanding of the kernel is essential.

Preparation Before Debugging

Identify a confirmed bug and its kernel version, use binary search to locate the introducing commit, ensure the bug is reproducible, and minimize the system to isolate variables.

Kernel Configuration Options for Debugging

Enable several kernel config options to gain extra debugging information:

Kernel hacking ---> [*] Magic SysRq key
[*] Kernel debugging
[*] Debug slab memory allocations
[*] Spinlock and rw‑lock debugging: basic checks
[*] Spinlock debugging: sleep‑inside‑spinlock checking
[*] Compile the kernel with debug info
[*] Driver Core verbose debug messages
[*] Load all symbols for debugging/ksymoops

Debugging Atomic Operations

Configure atomic‑operation debugging to warn when sleeping occurs inside atomic sections:

CONFIG_PREEMPT = y
CONFIG_DEBUG_KERNEL = y
CONFIG_KALLSYMS = y
CONFIG_SPINLOCK_SLEEP = y

BUG() and WARN() Macros

Use BUG() and BUG_ON() to trigger an OOPS with a stack trace; they act like assertions. WARN() and WARN_ON() print a stack trace without causing an OOPS.

#ifndef HAVE_ARCH_BUG
#define BUG() do { printk("BUG: failure at %s:%d/%s()! ", __FILE__, __LINE__, __FUNCTION__); panic("BUG!"); } while (0)
#endif
#ifndef HAVE_ARCH_BUG_ON
#define BUG_ON(condition) do { if (unlikely(condition)) BUG(); } while (0)
#endif

printk() Logging

printk()

can be called from any context (interrupt, process, locked sections) and supports log levels (KERN_EMERG … KERN_DEBUG). The default level is KERN_WARNING. Adjust /proc/sys/kernel/printk to control which messages appear on the console.

printk(KERN_CRIT "Hello, world!
");

Memory Debugging Tools

Several utilities help detect leaks and out‑of‑bounds accesses:

MEMWATCH : compile with -DMEMWATCH and include memwatch.h to log double‑free, leaks, and overruns.

YAMD : run with ./run-yamd <binary> to report multiple frees and leaks.

Electric Fence : links a malloc debugging library that aborts on fence‑post errors.

strace : traces system calls to pinpoint failures in user‑space programs.

OOPS Handling and ksymoops

An OOPS prints CPU registers, a stack trace, and a message indicating the faulting instruction. Use ksymoops with the OOPS log, the kernel System.map, and /proc/kallsyms to translate addresses to symbols.

Dynamic Debugging

Enable CONFIG_DYNAMIC_DEBUG to turn on/off pr_debug() / dev_debug() messages at runtime via /sys/kernel/debug/dynamic_debug/control using patterns like file foo.c line 123 +p.

kdump and kexec

kexec

loads a secondary “capture” kernel (the dump kernel) while the system is running. When a crash occurs, the capture kernel boots via kdump and writes a vmcore dump to disk.

Key steps:

Build a separate dump kernel or use a relocatable kernel as the capture kernel.

Reserve memory with the boot parameter crashkernel=128M (adjust size per architecture).

Configure /etc/kdump.conf (or /etc/sysconfig/kdump) to set dump location.

Enable the kdump service ( chkconfig kdump on && service kdump start).

Test by triggering a crash: echo c > /proc/sysrq-trigger.

Analyzing the Dump

After a crash, the vmcore file (e.g., /var/crash/vmcore) can be examined with the crash utility from the kernel-debuginfo package:

# crash /usr/lib/debug/lib/modules/$(uname -r)/vmlinux /var/crash/vmcore
crash> bt

These steps provide a reliable workflow for diagnosing kernel crashes and memory issues.