Master Linux Performance Monitoring: Essential Tools and How to Use Them

Running a Linux server generates a wealth of parameter data that is crucial for both operations staff and developers when diagnosing abnormal program behavior.

1. CPU and Memory Tools

1.1 top

The first line shows the 1‑, 5‑, and 15‑minute load averages; values exceeding the number of CPU cores indicate saturation. The second line lists task states (running, sleeping, stopped, zombie). The third line breaks down CPU usage:

(us) user : time spent in user mode with low nice values.

(sy) system : time spent in kernel mode.

(ni) nice : time spent in user mode with high nice values.

(id) idle : idle time.

(wa) iowait : time waiting for I/O.

(hi) irq : time handling hardware interrupts.

(si) softirq : time handling software interrupts.

(st) steal : time stolen by the hypervisor in virtualized environments.

High CPU usage suggests specific investigation paths, such as identifying offending processes, checking I/O intensity, adjusting nice values, or examining interrupt statistics.

1.2 vmstat

Vmstat provides a snapshot of processes, memory, and I/O. Columns include r (runnable processes), b (blocked), swpd (used swap), buff, cache, bi / bo (blocks I/O), in (interrupts), and cs (context switches).

1.3 pidstat

Pidstat offers per‑process statistics. Useful options: -t: show per‑thread details. -r: page faults (minor minflt/s and major majflt/s). -s: stack usage. -u: CPU usage. -w: thread context switches. -C pattern : filter by command name.

Pidstat is especially handy for multithreaded applications.

1.4 Other CPU Tools

Use mpstat -P ALL 1 to view per‑core load, top -u $USER or ps pipelines to filter processes, and ps axjf for a detailed process tree.

2. Disk I/O Tools

2.1 iostat

Running iostat -xz 1 (or sar -d 1) highlights key metrics:

avgqu-s : average queue length; >1 indicates saturation.

await / r_await / w_await : average I/O wait time.

svctm : average service time.

%util : device utilization; >60% degrades performance, near 100% means saturation.

Even with high I/O latency, kernel caching may mask impact on applications.

3. Network Tools

3.1 netstat

netstat -s

displays cumulative protocol statistics; netstat -antp and netstat -nltp list active TCP connections and listening sockets.

3.2 sar

Sar can monitor network activity with sar -n TCP,ETCP 1 (TCP) and sar -n UDP 1 (UDP), reporting metrics such as active connections, passive connections, retransmissions, and error rates.

3.3 tcpdump

Tcpdump captures packets for offline analysis with Wireshark. Use filters to limit capture scope and avoid excessive overhead; the -C / -W options rotate files when they reach a size limit.

These tools together provide a comprehensive view of system health, enabling precise identification of performance bottlenecks across CPU, memory, disk, and network layers.