Master Linux Server Performance: Essential Monitoring Tools & How to Interpret Their Metrics

Linux servers expose many parameters; operators and developers need to monitor them to diagnose issues.

1. CPU and Memory

The top command shows load averages, task states, and per‑CPU usage. The first line displays 1, 5, and 15‑minute load averages, indicating system load trends; values exceeding the number of CPU cores signal saturation.

The second line lists task states (running, sleeping, stopped, zombie). The third line breaks down CPU time into categories: (us) user: time spent in user space with low nice values. (sy) system: time spent in kernel space. (ni) nice: time spent in user space with high nice values. (id) idle: idle time. (wa) iowait: time waiting for I/O. (hi) irq: time handling hardware interrupts. (si) softirq: time handling software interrupts. (st) steal: time stolen by the hypervisor in virtualized environments.

High values in these columns suggest specific bottlenecks, and the article lists troubleshooting steps for each case.

1.2 vmstat

vmstat

reports runnable processes (r), uninterruptible sleep (b), virtual memory usage (swpd), buffers, cache, I/O blocks (bi/bo), interrupts (in), and context switches (cs).

1.3 pidstat

pidstat -t -C "pattern" -l

provides per‑process statistics such as page faults (minor minflt/s, major majflt/s), stack usage, CPU usage, and thread‑level context switches (active and passive).

1.4 Other tools

mpstat -P ALL 1

shows per‑core load to check load balancing. The ps command can be scripted to filter processes, e.g.:

while :; do ps -eo user,pid,ni,pri,pcpu,psr,comm | grep 'ailawd'; sleep 1; done

Process trees can be displayed with ps axjf.

2. Disk I/O

Tools like iotop and lsof reveal per‑process disk activity. iostat -xz 1 and sar -d 1 expose key metrics:

avgqu‑s: average queue length; values >1 may indicate saturation.

await (r_await, w_await): average I/O wait time.

svctm: average service time; close to await means little wait.

%util: device utilization; >60% may degrade performance, >100% indicates saturation.

3. Network

netstat -s

displays protocol statistics; combined with netstat -antp it lists listening sockets. sar -n TCP,ETCP 1 provides TCP counters such as active/s, passive/s, retrans/s, and isegerr/s. sar -n UDP 1 reports UDP counters like noport/s and idgmerr/s. tcpdump captures packets for offline analysis; using filters and size limits (-C, -W) keeps captures manageable.

These commands form a practical toolbox for Linux performance monitoring and troubleshooting.