This practical guide walks you through Linux kernel stack backtracing, covering GDB installation, core dump analysis, handling corrupted stacks, and advanced tracing techniques using ftrace and eBPF, with step‑by‑step commands, code examples, and troubleshooting tips to pinpoint the root cause of crashes.
This article explains the technical principles behind Linux kernel tracing, covering static tracepoints and dynamic kprobes, demonstrating their use with ftrace and perf, and detailing the underlying macro implementations and low‑level mechanisms that make kernel source tracking possible.
This article explains how to locate and monitor kernel execution using IDE navigation, Linux tracing tools such as ftrace, perf, tracepoints, and kprobes, and dives into the underlying implementation details of static and dynamic tracing mechanisms.
Discover how deep understanding of Linux kernel architecture, process, memory, filesystem, and network subsystems combined with practical Bash scripts can boost system performance by up to 300%, offering step‑by‑step tuning, monitoring, and debugging techniques essential for senior operations engineers.
This article explains the problem of internal and external memory fragmentation in Linux systems, introduces eBPF as a powerful tracing tool, and provides step‑by‑step guidance for building, loading, and running eBPF programs to analyze and mitigate fragmentation on both Linux and Android platforms.
The article details how a long‑standing bug that displayed incorrect call‑address information in bpftool’s JIT disassembly was reproduced, analyzed, and fixed by correcting the PC parameter to use the function’s kernel symbol address, with patches applied to both LLVM and libbfd back‑ends.
The article explains how to use eBPF tracepoints to monitor and record changes in Linux process capabilities, detailing the kernel data structures, BPF program logic, and user‑space handling needed to debug real‑world capability issues such as tcpdump failures and systemd service launches.
retsnoop is an eBPF‑based tracing utility that uses wildcard patterns to hook kernel functions, automatically captures full stack traces whenever a function returns an error, and offers three complementary modes—stack trace, function‑call trace, and LBR—to quickly pinpoint the source of kernel failures, with practical examples and source‑code insights.
Learn step‑by‑step how to capture a Linux core dump with kdump, install the crash utility and debuginfo packages, and use commands like sys, bt, ps, files, and sym to trace the root cause of kernel panics, illustrated with a real‑world case study.
This article explains how Linux system calls serve as the primary interface between user‑space programs and the kernel, describes the transition from user mode to kernel mode, and provides step‑by‑step debugging examples and source‑code snippets for x86_64 architectures.
The article explains how to use the Linux crash utility for efficient, on‑demand analysis of Android Kdump dumps on MTK and Qualcomm devices, covering dump loading parameters, baseline data collection, stack back‑tracing, local variable recovery, thread searching, and why crash outperforms memory‑heavy tools like trace32.
This article provides a comprehensive overview of Linux kernel debugging, covering core tools such as printk, ftrace, trace‑cmd, kprobe, systemtap, kgdb, kgtp, perf, as well as pseudo filesystems like procfs, sysfs, debugfs and relayfs, and introduces additional tracers including LTTng, eBPF, Ktap, dtrace4linux, OL DTrace and sysdig.
This guide explains why kernel debugging is challenging and walks through preparation steps, essential kernel configuration options, useful macros like BUG() and WARN(), logging with printk(), memory‑debugging utilities such as MEMWATCH, YAMD, Electric Fence and strace, handling OOPS messages, dynamic debugging, and setting up kdump/kexec for crash dumps.
The article explains Linux hung‑task detection, detailing both a system‑wide polling method that compares unchanged D‑state task context‑switch counts and a watchdog approach for critical processes, describes kernel implementations, analysis of real‑world hang cases, and emphasizes log analysis and parameter tuning to prevent system hangs.
This guide walks through using the Crash utility to parse a kdump vmcore, locate the sk_buff structure, examine its fields, and trace related socket data, illustrating a practical approach to diagnosing kernel memory corruption.
This guide shows how to set up an Ubuntu 16.04 virtual machine on Windows, install QEMU, fetch a modernized Linux 0.11 source tree, and use VSCode Remote‑SSH together with GDB to debug the historic kernel without rebuilding outdated toolchains.
This guide outlines six effective debugging techniques—including dtdiff, kernel device‑tree inspection, U‑Boot fdt commands, dtc decompilation, kernel fdt inspection, and of_property APIs—to help developers quickly locate and resolve device‑tree‑related issues during driver development.
Learn how to compile the Linux 4.19.172 kernel on CentOS, create a rootfs with BusyBox, and set up QEMU‑based debugging using GDB, including required kernel boot parameters, troubleshooting static linking errors, and optional Eclipse visual debugging.
This guide explains how to use the open‑source Crash utility to analyze Linux kernel vmcore dumps, covering supported dump formats, step‑by‑step installation from source, essential debugging parameters, common Crash commands, and building and using extension modules such as trace and gcore.
This article provides a comprehensive tutorial on configuring KGDB on Linux, covering target and host machine preparation, kernel recompilation with debugging options, module installation, boot‑loader adjustments, and detailed remote‑gdb debugging procedures for both the kernel image and loadable modules.
