Master Linux Kernel Stack Tracing: From GDB Basics to Advanced ftrace & eBPF

Linux kernel crashes often leave only fragmented core dumps or oops messages, making it hard to locate the fault. Kernel stack backtracing follows the call chain to reconstruct the execution flow at the moment of failure, revealing the exact code path that triggered the exception.

1. Preparation

1.1 What is kernel stack backtrace

A kernel stack backtrace records the call stack when a crash occurs, similar to a detective gathering clues at a crime scene. Each function call creates a stack frame that stores parameters, local variables, and the return address, allowing developers to see the sequence of calls leading to the crash.

1.2 Installing GDB

GDB (GNU Debugger) is the primary tool for debugging Linux programs. Install it with the package manager of your distribution.

sudo apt-get update

sudo apt-get install gdb

On Red Hat‑based systems: sudo dnf install gdb Verify the installation: gdb --version On Windows, install MinGW and ensure the GDB component is selected, then add the MinGW bin directory to PATH. Verify with:

gdb -v

1.3 Starting GDB

To debug an executable named test: gdb test To attach to a running process with PID 12345:

gdb -p 12345

2. Basic GDB Commands

Set breakpoint – pause execution at a specific location.

(gdb) b main

(gdb) b test.c:10

Run program – start execution (optionally with arguments). (gdb) run arg1 arg2 Step execution – next (skip over function calls) or step (enter functions).

(gdb) n

(gdb) s

Print variable – inspect values of variables or struct members.

(gdb) p num

(gdb) p p.x

Backtrace – display the call stack.

(gdb) bt

(gdb) bt full

(gdb) bt 2

3. Advanced Kernel Tracing

3.1 ftrace – static and dynamic instrumentation

ftrace inserts a call mcount at each function entry when CONFIG_FUNCTION_TRACER is enabled. Dynamic instrumentation replaces reserved NOP bytes with call mcount on demand, minimizing overhead.

Typical workflow to trace the open system call:

cd /sys/kernel/debug/tracing

echo '__x64_sys_open' > set_ftrace_filter

echo 'function' > current_tracer

ls /tmp > /dev/null

cat trace

Sample output shows process name, PID, CPU, timestamp, function name, and arguments.

3.2 eBPF – programmable tracing

eBPF programs are compiled to bytecode, verified for safety, JIT‑compiled in the kernel, and attached to tracepoints, kprobes, or other hooks. Example: count system calls per second per process.

bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm] = count(); } interval:s:1 { print(@); clear(@); }'

The script prints a per‑process syscall count every second.

3.3 Visualizing trace data

Large trace logs can be visualized with tools like KernelShark, which renders ftrace output as a timeline, highlighting hot paths and latency.

4. Practical Kernel Crash Analysis

4.1 Obtaining kernel stack information

Use gdb with the kernel image ( vmlinux) and a core dump ( vmcore), or use kdump to generate vmcore automatically on panic. gdb /path/to/vmlinux /path/to/vmcore Run bt to view the kernel call stack.

4.2 Interpreting the stack

Identify the function at #0 (crash point), examine parameters (e.g., file descriptor, buffer address), and trace upward through #1, #2, etc., to locate the offending code path.

4.3 Symbolization

Convert raw addresses to source locations with addr2line: addr2line -e my_program -f < stack_trace.txt This yields function names and line numbers, making the backtrace human‑readable.

4.4 Case study

A web server crashed with a segmentation fault in sys_read. The backtrace showed an invalid buffer pointer. Symbolization revealed the pointer originated from a faulty memory allocation in the server’s file‑reading routine. Fixing the allocation eliminated the crash.

5. Common Issues and Solutions

5.1 Incomplete stack traces

Ensure the kernel is built with CONFIG_DEBUG_INFO and CONFIG_FRAME_POINTER. Use memtest86+ to rule out hardware memory errors.

5.2 Symbolization failures

Make sure the correct symbol file matching the exact kernel or binary version is supplied to addr2line. Mismatched versions will produce unresolved addresses.

By following these steps—installing GDB, mastering basic debugging commands, leveraging ftrace and eBPF for dynamic tracing, and applying proper symbolization—you can efficiently locate and resolve kernel crashes.