Master Linux Performance: Essential Tools & Commands for System Monitoring

Driven by a strong interest in Linux and a desire to understand low‑level system behavior, this article serves as both a knowledge checklist and a practical guide to Linux performance analysis and optimization.

Performance Analysis Tools

The content is based on Brendan Gregg's Linux performance tuning blog and aggregates related articles to explain the principles and tools involved.

Background knowledge such as hardware caches and kernel behavior is essential because these low‑level factors can unexpectedly affect application performance, e.g., insufficient cache utilization or excessive system calls causing frequent kernel/user switches.

vmstat – Virtual Memory Statistics

vmstat monitors virtual memory, processes, CPU, and other system metrics. Typical usage: vmstat interval times Omit times to sample continuously until stopped (Ctrl+C).

Key columns include:

procs: r – processes waiting for CPU; b – processes in uninterruptible sleep (I/O).

memory: swap usage, free memory, buffers, cache.

swap: pages swapped in/out per second.

io: blocks read ( bi) and written ( bo).

system: interrupts ( in) and context switches ( cs).

cpu: percentages of user, system, idle, I/O wait, etc.

Symptoms of memory shortage include rapid drop in free memory, heavy swapping, increased I/O, higher interrupt and context‑switch counts, and more processes waiting on I/O.

iostat – CPU and I/O Statistics

iostat reports CPU usage and I/O statistics for devices, partitions, and network interfaces. It shows average values since boot and per‑device averages.

Common Linux disk I/O abbreviations: rrqm/s and wrqm/s: merged read/write requests per second. r/s and w/s: read/write requests per second. rsec/s and wsec/s: sectors read/written per second. avgrq‑sz: average request size. avgqu‑sz: average queue length. await: average wait time per I/O request. svctm: average service time. %util: percentage of time the device was busy.

dstat – System Monitoring

dstat displays CPU, disk I/O, network packets, and paging activity in a colorful, easy‑to‑read format, offering more detail than vmstat or iostat.

dstat -cdlmnpsy

iotop – Real‑time Disk I/O Monitoring

iotop shows per‑process I/O usage in a top‑like interface.

iotop –bod interval

pidstat – Process Resource Monitoring

pidstat reports CPU, memory, I/O, task switching, and thread statistics for all or selected processes.

pidstat –d interval

pidstat –u interval

pidstat –r interval

top and htop

top summarizes system load, process states, CPU usage, memory usage, and swap statistics, and lists detailed per‑process information.

htop is an interactive, color‑enhanced alternative to top, supporting mouse operation, horizontal/vertical scrolling, and easier process management.

mpstat

mpstat reports per‑CPU statistics from /proc/stat, useful on multiprocessor systems.

mpstat -P ALL interval times

netstat

netstat displays network statistics for IP, TCP, UDP, and ICMP, useful for checking open ports and routing tables.

netstat -npl   # show listening ports

netstat -rn    # display routing table

netstat -i     # interface statistics

ps

ps shows current process status; common usages include:

ps aux

ps -ef | grep

To kill a process:

ps aux | grep mysqld | grep -v grep | awk '{print $2}' | xargs kill -9

To kill zombie processes:

ps -eal | awk '{if ($2 == "Z") {print $4}}' | xargs kill -9

strace

strace traces system calls and signals of a program, helping diagnose runtime issues.

strace -e stat64 mysqld --print --defaults > /dev/null

uptime

uptime prints how long the system has been running and the load averages for the past 1, 5, and 15 minutes.

lsof

lsof lists open files and can be used to inspect file system blocks, ports, user‑opened files, and process‑opened files.

lsof /boot

lsof -i :3306

lsof -u username

lsof -p 4838

lsof -i @192.168.34.128

perf

perf is a kernel‑integrated performance analysis tool that can profile hot functions, cache misses, and other metrics, useful for both user‑space and kernel‑space optimization.

Sampling is typically driven by timer interrupts; high‑frequency sampling reveals where the program spends most of its time.

Summary

By combining the above commands with the introductory diagram, one can map performance aspects to the appropriate tools during analysis.

Common Performance Testing Tools

perf_events – kernel‑maintained performance diagnostics.

eBPF tools – BCC‑based tracing with user‑space map management.

perf-tools – lightweight suite based on perf_events and ftrace.

bcc (BPF Compiler Collection) – framework for creating efficient kernel tracing programs.

ktap – dynamic kernel tracing similar to DTrace/SystemTap.

Flame Graphs – visualizations generated from perf, SystemTap, or ktap data.

Linux Observability Tools

Basic tools: uptime, top/htop, mpstat, iostat, vmstat, free, ping, nicstat, dstat.

Advanced tools: sar, netstat, pidstat, strace, tcpdump, blktrace, iotop, slabtop, sysctl, /proc.

Linux Benchmarking Tools

Various tools exist for measuring performance of different subsystems; refer to attached documentation for details.

Linux Tuning Tools

These tools focus on kernel‑level tuning; see attached documentation for deeper exploration.

Linux Observability – sar

sar (System Activity Reporter) provides comprehensive system activity reports covering file I/O, system calls, disk I/O, CPU efficiency, memory usage, process activity, and IPC. sar [options] [-A] [-o file] t n Here t is the sampling interval, n the number of samples, and -o file writes results in binary format.