Essential Linux Performance Monitoring Tools and How to Use Them

The author compiled this guide out of a strong interest in Linux operating systems and a desire to understand low‑level performance characteristics, using Brendan Gregg’s performance‑tuning articles as a primary reference.

Performance Analysis Tools

vmstat – Virtual Memory Statistics

vmstat reports virtual memory, process, and CPU statistics. Typical usage: vmstat interval times (interval in seconds, optional repeat count). It continuously samples until stopped with Ctrl+C. The output columns include:

procs: r – runnable processes, b – blocked (uninterruptible sleep).

memory: swpd – swapped out pages, free – idle memory, buff – buffers, cache – page cache.

swap: si – pages swapped in, so – pages swapped out.

io: bi – blocks received, bo – blocks sent.

system: in – interrupts per second, cs – context switches per second.

cpu: us – user time, sy – system time, id – idle, wa – I/O wait.

Symptoms of memory pressure include rapid decline of free memory, high swap activity, increased I/O, and many blocked processes.

iostat – CPU and Device I/O Statistics

iostat shows CPU usage and per‑device I/O statistics. Use iostat -x for extended device details. Common abbreviations:

rrqm/s, wrqm/s – merged read/write requests per second.

r/s, w/s – read/write requests per second.

rsec/s, wsec/s – sectors read/written per second.

avgrq‑sz – average request size.

avgqu‑sz – average queue length.

await – average wait time per I/O.

svctm – average service time.

%util – percentage of time the device was busy.

dstat – System Monitoring Tool

dstat combines CPU, disk I/O, network, and paging statistics in a color‑coded, easy‑to‑read format. Example command:

dstat -cdlmnpsy

iotop – Real‑Time I/O Monitoring

iotop displays per‑process I/O usage similar to top. Non‑interactive usage: iotop -bod interval Related commands for detailed I/O statistics include pidstat -d interval.

pidstat – Process Resource Monitoring

pidstat can monitor CPU, I/O, and memory per process:

pidstat -d interval   # I/O

pidstat -u interval   # CPU

pidstat -r interval   # Memory

top and htop

top’s summary area shows load average, process states, CPU distribution, memory usage, and swap usage. The task area lists PID, user, priority, nice, virtual/physical memory, state, CPU%, MEM%, total CPU time, and command line. htop provides an interactive, color‑rich interface with mouse support, horizontal/vertical scrolling, and faster startup.

Scrollable process list with full command line.

Mouse interaction.

Quick process termination without typing PIDs.

mpstat – Multiprocessor Statistics

mpstat reports per‑CPU statistics from /proc/stat. Example:

mpstat -P ALL interval times

netstat – Network Statistics

Common usages:

netstat -npl   # show listening ports

netstat -rn   # display routing table

netstat -i    # interface statistics

ps – Process Status

Typical commands:

ps aux        # all processes

ps -ef | grep name

Kill a specific process:

ps aux | grep mysqld | grep -v grep | awk '{print $2}' | xargs kill -9

Kill zombie processes:

ps -eal | awk '{if ($2 == "Z") print $4}' | xargs kill -9

strace – System Call Tracing

Use strace to trace system calls and signals. Example to see which configuration file mysqld loads:

strace -e stat64 mysqld --print --defaults > /dev/null

uptime, lsof, perf

uptime prints how long the system has run and the 1‑, 5‑, 15‑minute load averages. lsof lists open files; useful for finding which process holds a file or port. perf is a kernel‑integrated profiling tool that samples functions, cache misses, and other events to locate performance hotspots.

Advanced Performance Testing Tools

perf_events – kernel‑maintained performance diagnostics.

eBPF tools – tracing via BCC, with maps managed from user space.

perf‑tools – lightweight collection based on perf_events and ftrace.

bcc (BPF Compiler Collection) – framework for creating eBPF programs and utilities.

ktap – dynamic kernel tracing script language, similar to DTrace/SystemTap.

Flame Graphs – visual representation of stack samples, generated with github.com/brendangregg/flamegraph.

Linux Observability Tools

Basic tools: uptime, top/htop, mpstat, iostat, vmstat, free, ping, nicstat, dstat.

Advanced tools: sar, netstat, pidstat, strace, tcpdump, blktrace, iotop, slabtop, sysctl, /proc.

Linux Benchmarking Tools

Various utilities exist for measuring performance of specific subsystems; refer to the attached documentation for detailed options.

Linux Tuning Tools

These tools focus on kernel‑level tuning, often requiring source‑code modifications or sysctl adjustments.

sar – System Activity Reporter

sar provides comprehensive system activity reports, covering I/O, CPU, memory, process, and IPC statistics. Typical invocation: sar [options] [-A] [-o file] interval count ‘interval’ is the sampling period in seconds, ‘count’ is the number of samples (default 1). The -o option saves binary output for later analysis.