Master Linux Performance Monitoring: Essential Tools and Metrics

Running a Linux server generates many parameters that are crucial for both operations staff and developers when troubleshooting abnormal program behavior.

1. CPU and Memory

1.1 top

The first line shows the 1, 5, and 15‑minute load averages; values exceeding the number of CPU cores indicate saturation. The second line lists task states (running, sleeping, stopped, zombie). The third line breaks down CPU usage by type: us (user), sy (system), ni (nice), id (idle), wa (iowait), hi (hardware IRQ), si (software IRQ), and st (steal, relevant for virtual machines).

High values in these fields suggest specific investigation paths, such as identifying CPU‑hungry processes, checking I/O intensity, examining nice values, or detecting excessive interrupt handling.

1.2 vmstat

vmstat

reports runnable processes (r), uninterruptible sleep (b), used swap (swpd), buffers, cached memory, I/O blocks (bi/bo), system interrupts (in), and context switches (cs). It helps correlate CPU load with memory pressure and I/O activity.

1.3 pidstat

pidstat -t -C "ailaw" -l

provides per‑process statistics, including page faults ( minflt/s, majflt/s), stack usage, CPU usage, and thread context switches. It is especially useful for detailed analysis of multithreaded applications.

1.4 Other CPU tools

For per‑CPU load balancing, mpstat -P ALL 1 shows each core’s utilization. Filtering top by user (e.g., top -u taozj) or using ps pipelines can isolate specific processes.

2. Disk I/O

2.1 iostat

Command: iostat -xz 1 Key metrics: avgqu-s (average queue length), await (average I/O wait), svctm (service time), and %util (device utilization). Values above thresholds indicate saturation.

3. Network

3.1 netstat

Typical usage: netstat -s for protocol statistics, netstat -antp for all TCP connections, netstat -nltp for listening sockets.

3.2 sar

Network monitoring: sar -n TCP,ETCP 1 and sar -n UDP 1 report active/passive connections, retransmissions, errors, and UDP port statistics.

3.3 tcpdump

tcpdump

captures packets for offline analysis with Wireshark. Use filters to limit capture size ( -C, -W) and to focus on specific hosts, ports, or protocols.