Master Linux Kernel Crash Analysis with Crash: Install, Commands & Extensions

What is Crash?

Crash is a utility that extends GDB to analyze Linux kernel core dump files (vmcore) generated by kexec/kdump, makedumpfile, Netdump, Diskdump, Xen, IBM 390, LKCD, Mcore, and other formats. It interprets the ELF dump and provides kernel‑specific commands.

Installation

Clone the source repository:

git clone git://github.com/crash-utility/crash.git

Install required development packages (example for Debian/Ubuntu): sudo apt-get install libncurses5-dev zlib1g-dev Build for the target architecture (example for ARM64):

cd crash-7.2.8
make target=ARM64

Strip symbols to reduce binary size (optional):

strip -s crash

Basic Debugging Usage

Crash requires two arguments:

NAMELIST – the uncompressed kernel image (vmlinux) containing symbols, usually provided by the kernel‑debug package.

MEMORY‑IMAGE – the ELF dump file (vmcore) to be analysed.

If the kernel was built with KASLR, supply the offset with --kaslr <offset> and any additional memory‑image options. Example command line:

./crash ../dump/vmlinux \
../dump/DDRCS0_0.BIN@0x80000000,../dump/DDRCS0_1.BIN@0x100000000 \
--kaslr <kaslr_offset> -m kimage_voffset=0xffffffe5d6200000 -m vabits_actual=39

Note: vabits_actual=39 corresponds to the kernel configuration CONFIG_ARM64_VA_BITS.

Common Crash Commands

Crash uses GDB 7.6 as its engine, so most GDB commands are available. Typical kernel‑specific commands include: log | tail -n 100 – show the last 100 lines of the kernel log that triggered the dump. ps | grep RU – list active processes on the CPU that generated the watchdog bark. struct msm_watchdog_data wdog_data – inspect the watchdog data structure.

Extension Commands

Crash can load shared‑object extensions that add new commands. Two useful extensions are trace and gcore.

trace

Exports ftrace data to a file:

trace show > FTRACE

gcore

Creates a core dump of a user‑space process (e.g., the init process). Example:

gcore -f 255 1

Building Extension Libraries

Compile trace‑cmd, which is required by the trace extension:

git clone git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/trace-cmd.git
cd trace-cmd
make

Copy trace.c into the Crash source extensions directory and build the extensions: make extensions The resulting trace.so will appear in the extensions folder.

Set the environment variable so Crash can locate the trace‑cmd binary: export TRACE_CMD=<path-to>/trace-cmd Load the extension inside Crash: extend <path-to>/trace.so After loading, the trace commands become available (e.g., trace show).

References

Crash whitepaper – http://people.redhat.com/anderson/

Crash analysis tutorial – https://www.dedoimedo.com/computers/crash-analyze.html

Crash source repository – https://github.com/crash-utility/crash