Common Linux Performance Analysis Tools and How to Use Them

Performance analysis tools

All tools can be queried via man for help pages.

vmstat – Virtual memory statistics

Syntax: vmstat interval times Samples every interval seconds for times iterations; omitting times runs continuously until interrupted (e.g., Ctrl+C).

procs : r – processes waiting for CPU; b – processes in uninterruptible sleep (I/O wait).

memory : swpd – swapped pages; other columns show free, buffered, and cached memory.

swap : pages swapped in and out per second.

io : bi (blocks read) and bo (blocks written), reflecting disk I/O.

system : interrupts ( in) and context switches ( cs) per second.

cpu : percentage of CPU time spent in user, system, idle, and I/O wait states.

iostat – CPU and I/O statistics

Reports CPU usage and I/O statistics for devices, adapters, and partitions. The first line shows averages since boot; subsequent lines show per‑device averages. rrqm/s and wrqm/s: merged read/write requests per second. r/s and w/s: read and write requests per second. rsec/s and wsec/s: sectors read/written per second. avgrq‑sz: average request size. avgqu‑sz: average queue length. await: average wait time per I/O request. svctm: average service time. %util: percentage of time the device was busy.

dstat – System monitoring

Displays CPU usage, disk I/O, network packets, and paging activity in colour, offering more detail than vmstat or iostat.

dstat -cdlmnpsy

iotop – Real‑time disk I/O monitor

Top‑like interface for per‑process I/O statistics. iotop -bod interval Non‑interactive mode can be combined with pidstat -d interval to list I/O per process.

pidstat – Per‑process resource monitoring

Reports CPU, memory, I/O, task switches, and thread activity for all or selected processes.

pidstat -d interval   # I/O statistics

pidstat -u interval   # CPU usage

pidstat -r interval   # Memory usage

top and htop – Process viewers

top shows a summary area with load average, process states, CPU breakdown (user, system, nice, idle, I/O wait, interrupts), memory totals, and swap usage, followed by a task area listing PID, user, priority, nice value, virtual/physical memory, state, CPU% and MEM%.

htop is an interactive, colour‑enhanced version of top that supports mouse operation, horizontal/vertical scrolling, and killing processes without typing a PID. Advantages over top include:

Scrollable process list with full command lines.

Faster startup.

Kill processes directly.

Mouse support.

mpstat – Multiprocessor statistics

Reads /proc/stat and reports per‑CPU or aggregate statistics.

mpstat -P ALL interval times

netstat – Network statistics

Displays IP, TCP, UDP, and ICMP statistics and is useful for checking local port usage.

netstat -npl   # show listening ports

netstat -rn   # print routing table

netstat -i    # interface statistics

ps – Process status

Common view: ps aux Find a specific process: ps -ef | grep process_name Kill a process by PID:

ps aux | grep mysqld | grep -v grep | awk '{print $2}' | xargs kill -9

Kill zombie processes:

ps -eal | awk '{if ($2 == "Z") {print $4}}' | xargs kill -9

strace – System call tracer

Records system calls and signals generated by a program, helping diagnose abnormal behaviour.

Example: show which configuration file mysqld loads.

strace -e stat64 mysqld -print -defaults > /dev/null

uptime – System uptime and load

Prints how long the system has been running and the average load over the past 1, 5, and 15 minutes.

lsof – List open files

Lists currently opened files and can be used to diagnose system issues.

Show files under /boot: lsof /boot Find which process uses a port (e.g., 3306): lsof -i :3306 List files opened by a user: lsof -u username Show files opened by a PID: lsof -p 4838 Show remote network connections from an IP:

lsof -i @192.168.34.128

perf – Kernel‑integrated performance tool

Bundled with the Linux kernel, perf can identify hot functions and cache‑miss ratios. It samples at tick interrupts, attributing each sample to the current execution context. If 90 % of a program’s runtime is spent in foo(), 90 % of samples will fall in foo() ’s context, allowing focused analysis.

sar – System Activity Reporter

Provides comprehensive system activity reports, covering file I/O, system calls, disk I/O, CPU efficiency, memory usage, process activity, and IPC. sar [options] [-A] [-o file] t n where t is the sampling interval, n the number of samples, and -o file stores results in binary format.