Mastering Linux KDUMP: Capture and Analyze Kernel Crashes Efficiently

Overview

KDUMP is a Linux kernel feature that captures a memory dump (vmcore) when the kernel panics, allowing post‑mortem analysis of the crash.

How KDUMP Works

KDUMP relies on the kexec mechanism to load a reserved crash kernel without a full reboot. The workflow consists of three stages:

Reserve crash kernel memory – a portion of RAM is set aside at boot time (e.g., crashkernel=128M).

Switch to the crash kernel – when the primary kernel panics, kexec boots the reserved kernel.

Generate the dump – the crash kernel runs kdump and writes vmcore to a configured location (local disk, NFS, etc.).

Configuring KDUMP

1. Install kexec tools

sudo yum install kexec-tools

2. Reserve memory for the crash kernel

Add the crashkernel parameter to the kernel command line, for example in /etc/default/grub: GRUB_CMDLINE_LINUX="crashkernel=128M" Rebuild the GRUB configuration and reboot:

sudo grub2-mkconfig -o /boot/grub2/grub.cfg
sudo reboot

3. Configure the dump destination

Edit /etc/kdump.conf and set the path directive to the directory that will store the dump files: path /var/crash Start and enable the service so that it runs automatically after a crash:

sudo systemctl start kdump
sudo systemctl enable kdump

Verifying the Setup

Trigger a manual crash to confirm that KDUMP captures a dump: echo c > /proc/sysrq-trigger The system will reboot, and a vmcore file will appear under the path defined in /etc/kdump.conf.

Analyzing the Dump with crash

Install the analysis tool

sudo yum install crash

Load the dump

Run crash with the uncompressed kernel image ( vmlinux) and the generated vmcore:

sudo crash /usr/lib/debug/lib/modules/$(uname -r)/vmlinux /var/crash/$(date +%Y-%m-%d-%H:%M)/vmcore

Common commands

log

– displays the kernel log leading up to the panic. bt – prints a backtrace of the crashing task.

Example output of log

crash> log
[   0.000000] Initializing cgroup subsys cpuset
[   0.000000] Initializing cgroup subsys cpu
...
[1183.443830] BUG: unable to handle kernel NULL pointer dereference at 0000000000000010
[1183.444069] IP: [<ffffffff810dffef>] __wake_up_common+0x2f/0x80
...

The log shows early kernel initialization, SELinux status changes, and finally the fatal NULL‑pointer dereference with the instruction pointer (IP) and module information.

Example output of bt

crash> bt
PID: 1234   TASK: ffff8801184c0000  CPU: 1   COMMAND: "echo"
#0 [ffff8801184c3a68] __wake_up_common at ffffffff810dffef
#1 [ffff8801184c3a98] __wake_up at ffffffff810e0038
#2 [ffff8801184c3ab8] complete at ffffffff810e3f20
#3 [ffff8801184c3ad8] i915_gem_object_set_to_gtt_domain at ffffffffa00c872f [i915]
...

The backtrace lists the call chain from the point of failure (#0) up to higher‑level functions, together with task information (PID, CPU, command) and register values useful for low‑level debugging.

Practical Applications

Server clusters – rapid identification of kernel crashes reduces downtime.

Embedded devices – captures rare crashes where interactive debugging is impossible.

Development and test labs – assists developers in debugging kernel modules and drivers.

Conclusion

When properly configured, KDUMP provides a reliable mechanism for capturing detailed kernel state after a panic. Combined with the crash utility, administrators and developers can analyze the dump, pinpoint the root cause, and restore system stability with minimal interruption.