Linux Troubleshooting Mastery: Tools, Flame Graphs & Real-World Case Study

Introduction: Troubleshooting Methodology

The article opens with a 5W2H framework for performance analysis: What (phenomenon), When (timing), Why (root cause), Where (location), How much (resource consumption), and How to do (resolution). It emphasizes that effective troubleshooting requires both experience and the right toolset to quickly pinpoint issues.

1. CPU Analysis

Key Concepts

Thread states are divided into on-CPU (executing, split into user and system time) and off-CPU (waiting for CPU, I/O, locks, paging). Understanding terms like processor, core, hardware thread, CPU cache, clock frequency, CPI/IPC, utilization, user/kernel time, scheduler, run queue, preemption, and multi-threading is essential.

Tools

uptime, vmstat, mpstat, top, pidstat — system-wide and per-process CPU/load overview.

perf — drills into per-function CPU usage inside a process, including kernel functions.

Usage Examples

// system CPU usage
top
// all CPU cores
mpstat -P ALL 1
// CPU stats and load average
vmstat 1
// per-process CPU stats
pidstat -u 1 -p pid
// function-level CPU profiling
perf top -p pid -e cpu-clock

2. Memory Analysis

Key Concepts

Covers main memory, virtual memory, resident memory, address space, OOM, page cache, page faults, swapping, swap space, user-space allocators (glibc, jemalloc), and kernel SLUB allocator.

Tools

free, vmstat, top, pidstat, pmap — memory usage statistics.

valgrind — memory leak detection.

dtrace — dynamic tracing with D scripts (requires deep kernel knowledge).

Usage Examples

// system memory
free -m
// virtual memory stats
vmstat 1
// process memory stats
pidstat -p pid -r 1
// process memory map
pmap -d pid
// memory leak check
valgrind --tool=memcheck --leak-check=full --log-file=./log.txt ./program

3. Disk I/O Analysis

Key Concepts

Disk is the slowest subsystem. Concepts include filesystem, VFS, page cache, buffer cache, directory cache, inode, inode cache, and noop scheduler.

Tools

iotop — per-process I/O.

iostat — detailed device I/O stats.

pidstat -d — per-process I/O.

perf record -e block:block_rq_issue — trace block I/O requests.

Usage Examples

iotop
iostat -d -x -k 1 10
pidstat -d 1 -p pid
perf record -e block:block_rq_issue -ag
perf report

4. Network Analysis

Key Concepts

Network monitoring is complex due to latency, blocking, collisions, packet loss, and external devices (routers, switches). Modern NICs are adaptive.

Tools

netstat, ss — connection states, socket summaries.

sar -n TCP,ETCP / DEV — TCP and device statistics.

tcpdump — packet capture.

tcpflow — stream-based capture.

Usage Examples

netstat -s
netstat -nu
netstat -apu
netstat -a | awk '/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}'
ss -t -a
ss -s
ss -u -a
sar -n TCP,ETCP 1
sar -n DEV 1
tcpdump -i eth1 host 192.168.1.1 and port 80
tcpflow -cp host 192.168.1.1

5. System Load Analysis

Key Concepts

Load measures work quantity (process queue length). Load Average is the 1/5/15-minute average.

Tools

uptime, top, vmstat — load overview.

strace -c -p pid — syscall time summary.

strace -T -e epoll_wait -p pid — trace specific syscall.

dmesg — kernel logs.

Usage Examples

uptime
top
vmstat
strace -c -p pid
strace -T -e epoll_wait -p pid
dmesg

6. Flame Graphs

Principles

Created by Brendan Gregg. Y-axis = call stack depth (top = running function). X-axis = sample count (width = time). Plateaus indicate hot spots. Colors are cosmetic (warm tones for CPU busy). Types: On-CPU, Off-CPU, Memory, Hot/Cold, Differential.

Dependencies

yum install systemtap systemtap-runtime
# kernel debuginfo must match kernel version (uname -r)
kernel-debuginfo-2.6.18-308.el5.x86_64.rpm
kernel-devel-2.6.18-308.el5.x86_64.rpm
kernel-debuginfo-common-2.6.18-308.el5.x86_64.rpm
debuginfo-install --enablerepo=debuginfo search kernel
debuginfo-install --enablerepo=debuginfo search glibc

Install quick_location Toolkit

git clone https://github.com/lidaohang/quick_location.git
cd quick_location

On-CPU Flame Graph (User & Kernel)

# on-CPU user
sh ngx_on_cpu_u.sh pid
cd ngx_on_cpu_u
# on-CPU kernel
sh ngx_on_cpu_k.sh pid
cd ngx_on_cpu_k
# serve SVG
python -m SimpleHTTPServer 8088
# open http://127.0.0.1:8088/pid.svg

Demo C program shows foo1 (1000 calls to foo3), foo2 (10 calls), main loop (1e9 iterations). The resulting flame graph clearly shows foo1 as the widest plateau.

Off-CPU Flame Graph

Analyzes why CPU utilization is low (waiting for I/O, locks, paging). Same script pattern with ngx_off_cpu_u.sh / ngx_off_cpu_k.sh.

Memory Flame Graph

sh ngx_on_memory.sh pid
cd ngx_on_memory
python -m SimpleHTTPServer 8088
# open http://127.0.0.1:8088/pid.svg

Differential Flame Graph (Red/Blue)

Compares two profiles (before/after code change). Red = increase, blue = decrease. Integrated into CI to catch performance regressions early.

cd quick_location
# profile before
perf record -F 99 -p pid -g -- sleep 30
perf script > out.stacks1
# profile after
perf record -F 99 -p pid -g -- sleep 30
perf script > out.stacks2
# generate diff
./FlameGraph/stackcollapse-perf.pl ../out.stacks1 > out.folded1
./FlameGraph/stackcollapse-perf.pl ../out.stacks2 > out.folded2
./FlameGraph/difffolded.pl out.folded1 out.folded2 | ./FlameGraph/flamegraph.pl > diff2.svg

Demo adds an add() function calling foo3 10,000 times per iteration; the diff graph highlights the new plateau in red.

7. Case Study: Nginx Cluster Anomaly

Symptom

On 2017-09-25 19:00, monitoring showed massive 499/5xx errors and sustained high CPU on nginx cluster.

Step-by-Step Analysis

Request traffic — traffic actually dropped, ruling out traffic spike.

Nginx response time — increased, pointing to nginx itself or upstream.

Upstream response time — also increased, suggesting upstream latency dragging nginx.

top — nginx worker processes consuming high CPU.

perf top -p pid — top consumers: free, malloc, JSON parsing.

On-CPU flame graph — revealed frequent JSON parsing with a low-performance library.

Root Cause & Resolution

The JSON parsing module was triggered per request. Upstream latency was actually caused by a loopback call back into nginx. The fix: disable the offending module (degrade). CPU dropped immediately and traffic normalized.

Key Takeaway

Prioritize known, clear bottlenecks (high CPU) over correlated symptoms (upstream latency). Flame graphs turned a vague "high CPU" into a precise function-level culprit within minutes.