Master Linux Performance: Essential Monitoring Tools and Practical Usage

Background

The author compiled this article out of a strong interest in Linux and a desire to master low‑level system knowledge. It serves both as a checklist for fundamental Linux concepts and as a reference for a wide range of performance‑analysis tools, assuming the reader already understands basic computer architecture, networking, and operating‑system principles.

Common Monitoring Commands

vmstat (Virtual Memory Statistics) reports overall system activity, including processes, memory, swap, I/O, system interrupts, and CPU usage. The basic syntax is vmstat interval times, where interval is the sampling period in seconds and times is the number of samples (omitting times runs indefinitely). The first line shows averages since boot; subsequent lines show real‑time data. Key columns include:

procs : r – runnable processes; b – blocked (uninterruptible sleep) processes.

memory : swpd – swapped memory; other columns show free, buffered, and cached memory.

swap : pages swapped in/out per second.

io : blocks read ( bi) and written ( bo).

system : interrupts ( in) and context switches ( cs).

cpu : percentages of user, system, idle, and I/O‑wait time.

Typical signs of memory pressure include a rapid drop in free memory, high swap activity, increased I/O, and many processes waiting on I/O.

iostat reports CPU statistics and detailed device I/O metrics. Use iostat -x to display extended device statistics, such as request merges ( rrqm/s, wrqm/s), reads/writes per second, sectors transferred, average request size, average queue length, await time, service time, and utilization ( %util).

dstat combines CPU, disk I/O, network, and paging statistics into a colorful, easy‑to‑read output. A typical invocation is dstat -cdlmnpsy, which shows CPU, disk, load, memory, network, and system statistics.

Process‑Level Monitoring

iotop displays real‑time disk I/O per process, similar to top. Run non‑interactively with iotop -bod interval to log I/O activity.

pidstat monitors resource usage of specific processes. Examples: pidstat -d interval – I/O statistics. pidstat -u interval – CPU usage. pidstat -r interval – memory usage.

top and htop provide interactive process views. top shows load average, process states, CPU breakdown, memory usage, and swap. htop adds color, mouse support, horizontal/vertical scrolling, and easier process killing.

mpstat (Multiprocessor Statistics) reports per‑CPU usage. Use mpstat -P ALL interval times to see each CPU’s statistics.

netstat displays network connections, routing tables, and interface statistics. Common usages: netstat -npl – list listening ports. netstat -rn – show routing table. netstat -i – interface statistics.

ps lists processes; typical commands include ps aux and ps -ef | grep <em>process_name</em>. To kill a process by PID:

ps aux | grep mysqld | grep -v grep | awk '{print $2}' | xargs kill -9

. To kill zombie processes: ps -eal | awk '{if ($2 == "Z") print $4}' | xargs kill -9.

strace traces system calls and signals of a program, useful for debugging. Example: strace -e stat64 mysqld -print -defaults > /dev/null.

uptime shows how long the system has been running and the load averages for the past 1, 5, and 15 minutes.

lsof lists open files. Useful forms include lsof /boot (files on /boot), lsof -i :3306 (process using port 3306), lsof -u username (files opened by a user), and lsof -p 4838 (files opened by PID 4838).

Advanced Profiling Tools

perf is a kernel‑integrated profiling suite that can sample CPU cycles, cache misses, and function hotspots. It works by setting up tick‑based sampling; the more samples collected from a hot function, the more confidence you have that it is a performance bottleneck.

References for deeper perf usage include the blog at http://blog.csdn.net/trochiluses/article/details/10261339.

sar (System Activity Reporter) is one of the most comprehensive Linux performance tools, reporting CPU, memory, I/O, process, and IPC activity. Basic syntax: sar [options] [-A] [-o file] t [n], where t is the interval and n the number of samples. The -o option writes binary output to file for later analysis.

Additional Resources

Other notable utilities mentioned are eBPF tools , bcc , perf‑tools , ktap , and Flame Graphs . Their source repositories include:

https://github.com/brendangregg/perf-tools

https://github.com/iovisor/bcc#tools

https://github.com/ktap/ktap

https://github.com/brendangregg/flamegraph

These tools enable kernel‑level tracing, custom eBPF programs, and visual performance analysis.

Illustrations