Comprehensive Guide to Linux Performance Analysis Tools: vmstat, iostat, perf, and More

Performance Analysis Tools Overview

The article begins by referencing Brendan Gregg's performance analysis tool diagram, noting all tools can be explored via man pages. It then provides detailed coverage of each tool with practical examples.

vmstat -- Virtual Memory Statistics

vmstat

monitors virtual memory, processes, and CPU. Standard usage: vmstat interval times (e.g., vmstat 5 samples every 5 seconds). The first line shows averages since boot; subsequent lines show current intervals. Key columns:

procs : r = processes waiting for CPU; b = processes in uninterruptible sleep (waiting for I/O)

memory : swpd = swapped to disk; free = unused; buff = buffer; cache = OS cache

swap : si = swapped in/sec; so = swapped out/sec

io : bi = blocks read from block device; bo = blocks written

system : in = interrupts/sec; cs = context switches/sec

cpu : us = user time; sy = system time; id = idle; wa = I/O wait

Memory pressure signs: free memory drops sharply, buffer/cache reclamation fails, heavy swap usage (swpd), frequent paging (si/so), increased disk I/O (bi/bo), rising interrupts (in), context switches (cs), more processes waiting for I/O (b), high CPU I/O wait (wa).

iostat -- CPU and I/O Statistics

iostat

reports CPU and I/O stats for devices, adapters, tty, disks, CD-ROM. Default shows CPU like vmstat; use iostat -x for extended device stats. First line = boot averages; subsequent lines = incremental averages per device. Common abbreviations: rq=request, r=read, w=write, qu=queue, sz=size, a=average, tm=time, svc=service. Key metrics:

rrqm/s, wrqm/s = merged read/write requests/sec

r/s, w/s = read/write requests sent to device/sec

rsec/s, wsec/s = sectors read/written/sec

avgrq-sz = average request size (sectors)

avgqu-sz = average queue length

await = average I/O request time (ms)

svctm = average service time (ms)

%util = percentage of time device had active requests

dstat -- Versatile System Monitoring

dstat

combines CPU, disk I/O, network, paging in colored, readable output. Run dstat for default view; dstat -cdlmnpsy for specific modules (CPU, disk, load, memory, network, paging, system, CPU).

iotop -- Per-Process Disk I/O Monitoring

iotop

shows which processes generate disk I/O, with top-like UI (PID, user, I/O, process). Non-interactive mode: iotop -bod interval. For per-process I/O: pidstat -d interval.

pidstat -- Per-Process Resource Monitoring

pidstat

monitors CPU, memory, device I/O, task switching, threads for all or specific processes. Usage examples:

I/O: pidstat -d interval CPU: pidstat -u interval Memory:

pidstat -r interval

top -- Classic Process Viewer

Summary area shows five categories: load (time, users, 1/5/15-min load), tasks (running, sleeping, stopped, zombie), CPU (us, sy, ni, id, wa, hi, si), memory (total, used, free, buffers, cache), swap (total, used, free). Task area default columns: PID, USER, PR, NI, VIRT, RES, SHR, S, %CPU, %MEM, TIME+, COMMAND.

htop -- Interactive Process Viewer

htop

(ncurses-based) advantages over top: horizontal/vertical scrolling for full command lines, faster startup, kill without typing PID, mouse support, color themes.

mpstat -- Multiprocessor Statistics

mpstat

reports per-CPU stats from /proc/stat. Common usage: mpstat -P ALL interval times shows all CPUs.

netstat -- Network Statistics

Displays IP, TCP, UDP, ICMP stats; checks port connections. Common commands: netstat -npl -- check if port is open netstat -rn -- print routing table netstat -in -- interface info (MTU, packets, errors, collisions, queue length)

ps -- Process Status

Many options; refer to man ps. Common: ps aux, ps -ef | grep pattern. Kill process:

ps aux | grep mysqld | grep -v grep | awk '{print $2}' | xargs kill -9

. Kill zombies: ps -eal | awk '{if ($2 == "Z"){print $4}}' | xargs kill -9.

strace -- System Call Tracing

Traces system calls and signals for debugging. Example: find which config file mysqld loads: strace -e stat64 mysqld --print-defaults > /dev/null.

uptime -- System Uptime and Load

Shows uptime and 1/5/15-minute load averages.

lsof -- List Open Files

Lists open files for diagnostics. Examples:

Filesystem block: lsof /boot Port occupancy: lsof -i :3306 User files: lsof -u username Process files: lsof -p 4838 Remote connections:

lsof [email protected]

perf -- Kernel Performance Profiling

perf

(Linux kernel built-in) leverages tight kernel integration for hotspot functions, cache miss rates. Principle: sampling (e.g., tick-based). If 90% time in foo(), ~90% samples land in foo() given sufficient frequency/duration.

Advanced Performance Tool Ecosystem

perf_events : kernel-maintained profiling for apps and kernel

eBPF tools : bcc-based tracing; eBPF maps managed in user space via bpf syscall

perf-tools : collection based on perf_events and ftrace; minimal deps; Linux 3.2+

bcc (BPF Compiler Collection) : eBPF-based perf tools; kernel 3.15+ (eBPF), 4.1+ (multi-purpose)

ktap : scriptable dynamic kernel tracing; similar to DTrace/SystemTap

Flame Graphs : visualization from perf/systemtap/ktap; source at

github.com/brendangregg/flamegraph

Tool Classification by Maturity

Linux Observability Tools (Basic)

uptime, top/htop, mpstat, iostat, vmstat, free, ping, nicstat, dstat

Advanced Observability Tools

sar, netstat, pidstat, strace, tcpdump, blktrace, iotop, slabtop, sysctl, /proc

Linux Benchmarking Tools

Various tools per module (image referenced).

Linux Tuning Tools

Kernel-source-level tuning (refer to attached docs).

sar -- System Activity Reporter

Comprehensive reporting: file I/O, syscalls, disk I/O, CPU, memory, processes, IPC. Usage: sar [options] [-A] [-o file] t [n] where t=interval, n=count (default 1), -o=binary output file.