Mastering KGDB: A Step‑by‑Step Guide to Kernel Debugging on Multi‑Core Linux

What is KGDB?

KGDB is the official Linux kernel debugger that uses a remote GDB connection to allow source‑level debugging of the kernel.

Advantages over printk and ftrace

KGDB provides breakpoints, register inspection, and single‑step execution, which printk and ftrace cannot offer, especially for multi‑core race conditions.

KGDB Architecture

Two machines are involved: the target machine runs the kernel with a KGDB stub, and the host machine runs GDB. The stub communicates over serial or Ethernet using the GDB remote protocol.

Installation and Configuration

Hardware requirements: development host (x86_64, ≥2 CPU, 4 GB RAM) with GDB 7.12+, kernel source, patch tools; target machine with matching architecture and a kernel compiled with KGDB support.

Prepare communication link: either a crossover RS‑232 cable (TXD↔RXD, GND) or Ethernet with an open port (e.g., 1234).

Apply the appropriate KGDB patch series to the kernel source (example for 2.6.7).

Enable KGDB options in make menuconfig (CONFIG_KGDB, CONFIG_KGDB_SERIAL_CONSOLE, CONFIG_DEBUG_INFO, etc.).

Compile the kernel with debug symbols (use -O instead of -O2 to avoid code reordering).

Install the kernel, initrd, and copy them to the target’s /boot directory.

Add KGDB boot parameters (e.g., kgdbwait kgdboc=ttyS0,115200 for serial or kgdboc=eth0,192.168.1.100:1234 for Ethernet) to GRUB and update the bootloader.

Running a Debug Session

On the host, start GDB with the kernel image: gdb ./vmlinux. Connect to the target using target remote /dev/ttyS0 or target remote 127.0.0.1:1234. Set breakpoints (e.g., b netif_receive_skb) and use standard GDB commands ( c, n, s, p, bt, info registers) to inspect state.

Common Pitfalls and Tips

Ensure the kernel is built with CONFIG_DEBUG_INFO and that the vmlinux file matches the target.

Use noinline attributes or compile with -fno-inline if breakpoints are missed due to inlining.

Load symbols for dynamically loaded modules with add-symbol-file or lx-symbols.

Avoid serial‑port conflicts by freeing the device from other programs.

Trigger a manual KGDB session with the Magic SysRq key ( Alt+SysRq+g) when kgdbwait is not set.