Unlocking Linux Kernel Secrets: A Comprehensive Guide to Debugging Tools

1 Kernel Debugging and Tool Summary

Kernel debugging is far more elusive than user‑space debugging, but the kernel developers provide a rich set of tools and subsystems to support it.

2 User‑Space and Kernel‑Space Data Exchange Filesystems

The kernel offers three common pseudo‑filesystems: procfs, sysfs and debugfs.

These filesystems enable data exchange between kernel and user space, each suited to different scenarios: procfs – the oldest, provides read‑only access to processor, memory, device driver and process information. sysfs – tightly coupled with the kobject framework, primarily serves device drivers. debugfs – created for debugging, more flexible than the others. relayfs – a fast relay filesystem for transferring large amounts of data from kernel to user space.

Related links:

In Linux, the ways to exchange data between user space and kernel space (Part 2) – procfs, debugfs and relayfs: http://www.ibm.com/developerworks/cn/linux/l-kerns-usrs2/

2.1 procfs

procfs

is an older mechanism that exports many kernel parameters via /proc. Most of its entries are read‑only, and many applications rely on it.

mkdir -p /sysfs
mount -t sysfs sysfs /sysfs

2.2 sysfs

sysfs

is a memory‑based filesystem built on ramfs. It exposes kernel data structures and their attributes to user space through the kobject subsystem.

2.3 debugfs

Developers often need to output debugging information that printk cannot handle efficiently. debugfs provides a small virtual filesystem for creating files that deliver debug data.

2.4 relayfs

relayfs

offers a high‑performance channel‑based mechanism. Each channel consists of per‑CPU kernel buffers; user‑space reads data via standard file I/O (open, mmap, poll, close).

2.5 seq_file

The seq_file interface improves large output handling in procfs, avoiding the need for multiple reads.

3 printk

The simplest kernel debugging method is printk, analogous to printf in user space, but it lacks fine‑grained control.

4 ftrace & trace‑cmd

4.1 ftrace

ftrace is the most powerful tracing framework in modern Linux, providing static and dynamic probe points.

4.2 trace‑cmd

trace‑cmd is a front‑end for ftrace, wrapping the /sys/kernel/debug/tracing filesystem.

# Collect information
sudo trace-cmd record subsystem:tracing
# Parse results
# sudo trace-cmd report

5 Kprobe & systemtap

5.1 Kprobe

Kprobe is a lightweight kernel debugging facility that allows insertion of probes at arbitrary instruction addresses.

Three probe types exist: kprobes, jprobes (entry probes) and kretprobes (return probes).

5.2 systemtap

SystemTap provides dynamic tracing using the Kprobe API, similar to Solaris DTrace, allowing scripts to define probes and actions.

6 kgdb & kgtp

6.1 kgdb

kgdb merges the earlier KDB and KGDB projects, offering remote debugging via GDB monitor commands.

(gdb) gdb monitor ps -A

6.2 kgtp

KGTP is a lightweight, real‑time kernel debugger and tracer that works without patching the kernel; simply load the KGTP module.

7 perf

Perf is a versatile performance analysis tool that uses PMU events, tracepoints, and software counters to profile both user‑space applications and the kernel.

It can compute IPC, sample functions, replace strace, and benchmark schedulers.

8 Other Tracer Tools

8.1 LTTng

LTTng is an open‑source Linux tracing framework consisting of kernel, user‑space, and control components.

8.2 eBPF

eBPF is a JIT‑compiled in‑kernel virtual machine that can run programs on events, extending tracing capabilities.

8.3 Ktap

Ktap uses an in‑kernel Lua VM for tracing on embedded devices, now being superseded by eBPF.

8.4 dtrace4linux

dtrace4linux is a Linux port of Solaris DTrace, offering experimental providers under the CDDL license.

8.5 OL DTrace

Oracle Linux DTrace brings DTrace to Oracle Linux with providers such as syscall, profile, sdt, proc, sched, and USDT.

8.6 sysdig

sysdig is a tcpdump‑style tracer that captures system calls and can be extended with Lua scripts; it is adding container and eBPF support.

Related links:

Various links to original articles and tutorials are retained throughout the text.