Master Linux Kernel Debugging: Tools, Filesystems, and Tracing Techniques

1 Kernel Debugging and Tool Summary

Kernel debugging is far more complex than user‑space debugging; the kernel provides a series of tools and subsystems to support debugging.

Debugging essentially involves data exchange between kernel space and user space, and the kernel offers various mechanisms to achieve this.

2 User‑Space and Kernel‑Space Data Exchange Filesystems

The kernel provides three common pseudo filesystems: procfs, sysfs and debugfs.

All are used for kernel‑user data exchange, but each has different usage scenarios. procfs – the oldest, originally the only way to interact with the kernel, exposing processor, memory, device driver and process information. sysfs – tightly coupled with the kobject framework; it serves device drivers. debugfs – named for debugging; more flexible for complex data structures. relayfs – a fast relay filesystem for transferring large amounts of data from kernel to user space.

Reference links:

In Linux the ways of user‑space/kernel‑space data exchange, part 2: procfs, seq_file, debugfs and relayfs – http://www.ibm.com/developerworks/cn/linux/l-kerns-usrs2/

Linux filesystems: procfs, sysfs, debugfs usage – http://www.tinylab.org/show-the-usage-of-procfs-sysfs-debugfs/

2.1 procfs Filesystem

ProcFs

introduction procfs is an older mechanism for kernel‑user data exchange; many kernel parameters are exported under /proc. Most of the data is read‑only, and many applications heavily rely on it.

Reference: 用户空间与内核空间数据交换的方式(2)——procfs – http://www.cnblogs.com/hoys/archive/2011/04/10/2011141.html

2.2 sysfs Filesystem

sysfs is a memory‑based filesystem built on ramfs. It exposes kernel data structures, their attributes, and the relationship to kobject, allowing user‑space programs to read and set kernel parameters.

mkdir -p /sysfs
mount -t sysfs sysfs /sysfs

Note: do not confuse sysfs with sysctl. sysctl controls kernel behavior, while sysfs exposes the kobject hierarchy.

Reference links:

用户空间与内核空间数据交换的方式(6)——sysfs – http://www.cnblogs.com/hoys/archive/2011/04/10/2011470.html

2.3 debugfs Filesystem

debugfs is a small virtual filesystem dedicated to exposing debugging information. It is more flexible than sysfs for complex data structures and is widely used by kernel developers.

Reference links:

用户空间与内核空间数据交换的方式(1)——debugfs – http://www.cnblogs.com/hoys/archive/2011/04/10/2011124.html

3 printk

In kernel debugging, the simplest method is using printk, which works like printf in user‑space programs but uses the kernel’s own library.

Reference links:

Linux kernel debugging technique – printk – http://www.cnblogs.com/veryStrong/p/6218383.html

4 ftrace & trace‑cmd

4.1 trace & ftrace

ftrace is the most powerful current debugging and tracing mechanism in Linux. It provides static and dynamic probe points to capture kernel behavior.

Static probe points are compiled into the kernel; dynamic points use the mcount mechanism to replace placeholder bytes at runtime.

ftrace helps developers understand kernel runtime behavior for fault debugging or performance analysis.

Reference links:

ftrace and its front‑end tool trace‑cmd – http://blog.yufeng.info/archives/1012

4.2 trace‑cmd Front‑End Tool

trace‑cmd introduction

trace‑cmd and the graphical tool kernelshark are front‑ends for ftrace, providing easier access to /sys/kernel/debug/tracing.

# collect information
sudo trace-cmd record subsystem:tracing
# parse results
sudo trace-cmd report

Reference links:

Trace‑cmd: A front‑end for Ftrace – https://lwn.net/Articles/410200/

5 Kprobe & systemtap

5.1 Kernel kprobe Mechanism

kprobe

is a lightweight kernel debugging facility that allows inserting probes at arbitrary kernel locations. It underlies higher‑level tools such as perf and systemtap.

kprobe supports three probe types: kprobe – can be inserted at any instruction. jprobe – can be inserted only at function entry. kretprobe – fires on function return.

Reference links:

kprobe working principle – http://blog.itpub.net/15480802/viewspace-1162094/

5.2 systemtap

SystemTap provides dynamic tracing of a running Linux kernel using the kprobe API. Scripts are written in a C‑like language to specify probes and actions.

SystemTap is similar to Solaris DTrace and has been ported to other platforms.

Reference links:

SystemTap learning notes – https://segmentfault.com/a/1190000000671438

6 kgdb & kgtp

6.1 kgdb

kgdb merges KDB and KGDB, offering a kernel debugger that can be accessed via GDB monitor commands, e.g., (gdb) gdb monitor ps -A. After merging, some KDB features such as assembly‑level source debugging are lost.

Reference links:

kprobe work principle – http://blog.itpub.net/15480802/viewspace-1162094/

6.2 KGTP

KGTP is a lightweight real‑time kernel debugger/tracer that works without patching the kernel; simply load the KGTP module and use it from GDB. It supports x86‑32, x86‑64, MIPS and ARM, and can be used on Android.

Reference links:

github‑KGTP – https://github.com/teawater/kgtp

7 perf

perf is a performance analysis tool that uses PMU, tracepoints and kernel counters. It can measure hardware events (e.g., instructions retired, CPU cycles) and software events (e.g., page faults, context switches), perform per‑thread analysis, function‑level sampling, replace strace, add dynamic probes, and benchmark the scheduler.

perf is often described as a Swiss‑army knife for performance analysis.

Reference links:

Perf – Linux system performance tuning, part 1 – https://www.ibm.com/developerworks/cn/linux/l-cn-perf1/index.html

8 Other Tracer Tools

8.1 LTTng

LTTng is an open‑source Linux tracing framework that can trace both kernel and user programs, managing trace sessions (start/stop, enable/disable events, etc.).

Reference links:

Linux platform open‑source tracing tool LTTng – http://www.open-open.com/lib/view/open1413946397247.html

8.2 eBPF

eBPF (extended Berkeley Packet Filter) is a high‑performance in‑kernel virtual machine that can run programs on events. It provides a foundation for ftrace and perf_events and enables advanced tracing such as block I/O latency heatmaps.

Reference links:

eBPF slides and samples – Alexei Starovoitov’s resources.

8.3 Ktap

Ktap used a Lua VM inside the kernel for tracing on embedded devices. It has been largely superseded by eBPF.

Reference links:

Ktap project – historical information.

8.4 dtrace4linux

dtrace4linux is a Linux port of Sun DTrace, experimental and not fully integrated into the kernel.

Reference links:

dtrace4linux – http://code.google.com/p/dtrace4linux/

8.5 OL DTrace

Oracle Linux DTrace brings DTrace to Oracle Linux, providing providers such as syscall, profile, sdt, proc, sched and USDT. Completion of the function‑boundary tracing (fbt) provider is awaited.

Reference links:

Oracle Linux DTrace – official documentation.

8.6 sysdig

sysdig is a tracer that uses a tcpdump‑like syntax and Lua scripts to capture system calls and events. It currently focuses on system‑call tracing and is adding container support.

Reference links:

sysdig project – https://github.com/draios/sysdig

Original author: CHENG Jian Original link: https://kernel.blog.csdn.net/article/details/68948080