Master Linux Performance: Essential Monitoring Tools Explained

Performance Analysis Tools

Motivated by a strong interest in Linux and a desire to understand low‑level system behavior, this guide collects the most commonly used performance‑analysis commands, providing a quick reference for system monitoring, troubleshooting, and optimization.

vmstat – Virtual Memory Statistics

vmstat reports overall system activity, including virtual memory, processes, CPU, and I/O. The basic syntax is vmstat interval times, where interval is the sampling period in seconds and times is the number of samples (omitted means run until interrupted).

The first line shows averages since boot; subsequent lines show current activity at each interval. Columns include:

procs: r – processes waiting for CPU, b – processes in uninterruptible sleep (I/O wait).

memory: swpd – swapped out pages, free – free memory, buff – buffers, cache – cache.

swap: pages swapped in/out per second.

io: bi – blocks received, bo – blocks sent.

system: interrupts ( in ) and context switches ( cs ).

cpu: percentages of time spent in user, system, idle, and I/O wait.

Low memory manifests as a rapid drop in free memory, heavy swapping, increased I/O, more page faults, and higher CPU time waiting for I/O.

iostat – CPU and I/O Statistics

iostat reports CPU usage and per‑device I/O statistics. The default output mirrors vmstat’s CPU section; adding device options shows extended metrics such as rrqm/s , wrqm/s , r/s , w/s , await , and %util .

dstat – Enhanced System Monitoring

dstat combines the information of vmstat, iostat, and netstat into a colorful, easy‑to‑read display. It shows CPU usage, disk I/O, network packets, and paging activity. A typical invocation is:

dstat -cdlmnpsy

iotop – Real‑Time Disk I/O Monitor

iotop displays per‑process I/O usage in a top‑like interface. It can also be run non‑interactively: iotop -bod interval To list I/O per process:

pidstat -d interval

pidstat – Per‑Process Resource Statistics

pidstat reports CPU, memory, I/O, and context‑switch activity for selected processes. Examples:

pidstat -d interval

pidstat -u interval

pidstat -r interval

top / htop – Interactive Process Viewers

top provides a summary of system load, process states, CPU usage, memory, and swap. htop adds color, mouse support, and easier navigation.

mpstat – Multiprocessor Statistics

mpstat reports per‑CPU utilization and can display statistics for all CPUs or a specific one.

mpstat -P ALL interval times

netstat – Network Connections and Routing

netstat shows IP, TCP, UDP, and ICMP statistics, useful for checking open ports and routing tables.

netstat -npl   # show listening ports
netstat -rn    # display routing table
netstat -i     # interface statistics

ps – Process Status

ps provides detailed information about running processes. Common usages:

ps aux
ps -ef | grep <em>process_name</em>

To kill a process by name:

ps aux | grep mysqld | grep -v grep | awk '{print $2}' | xargs kill -9

strace – System Call Tracer

strace traces system calls and signals of a program, helping to diagnose runtime errors.

strace -e stat64 mysqld --print --defaults > /dev/null

uptime – System Load Summary

uptime prints how long the system has been running and the average load over the past 1, 5, and 15 minutes.

lsof – List Open Files

lsof enumerates files opened by processes, useful for finding which process holds a file or port.

lsof /boot
lsof -i :3306
lsof -u username
lsof -p 4838
lsof -i @192.168.34.128

perf – Kernel‑Level Performance Analyzer

perf is a built‑in Linux kernel tool for profiling functions, cache misses, and other low‑level events. It samples the program at timer ticks to identify hot spots.

sar – System Activity Reporter

sar collects and reports a wide range of system metrics, including CPU, memory, I/O, and network activity.

sar [options] [-A] [-o file] t [n]

Basic vs. Advanced Tools

Basic tools (uptime, top/htop, mpstat, iostat, vmstat, free, ping, nicstat, dstat) give a quick overview of system health. Advanced tools (sar, netstat, pidstat, strace, tcpdump, blktrace, iotop, slabtop, sysctl, /proc) provide deeper insight for detailed troubleshooting.

Observability, Benchmarking, and Tuning

Observability tools such as perf_events, eBPF, perf‑tools, bcc, and ktap enable fine‑grained tracing and visualization (e.g., Flame Graphs). Benchmarking utilities evaluate performance of specific subsystems, while tuning tools modify kernel parameters to improve overall efficiency.

By combining these commands with the diagrams above, readers can quickly identify which tool to use for a given performance problem.