Essential Linux Server Monitoring Tools and How to Interpret Their Metrics

Linux servers constantly expose a wealth of performance parameters that are crucial for both operations engineers and developers when troubleshooting abnormal program behavior.

Simple command‑line tools read data from

/proc

and

/sys

to present these metrics; more advanced analysis may require specialized utilities such as

perf

or

systemtap

.

CPU and Memory Monitoring

top

top

displays load averages, task states, and per‑CPU usage. The first line shows 1‑, 5‑, and 15‑minute load averages; values exceeding the number of CPU cores indicate saturation. The second line lists task counts (running, sleeping, stopped, zombie). Subsequent columns break down CPU time into user (us), system (sy), nice (ni), idle (id), iowait (wa), irq (hi), softirq (si), and steal (st). High values in each column suggest specific investigation paths, such as locating CPU‑intensive processes, checking I/O‑bound kernel activity, or detecting hypervisor over‑commit.

The fourth and fifth lines report physical and virtual memory.

total = free + used + buff/cache

;

Buffers

cache raw disk metadata, while

Cached

stores file data.

Avail Mem

indicates memory readily usable without swapping. Frequent swap activity signals memory pressure.

Note that

top

itself consumes resources and is best for short‑term, interactive monitoring.

vmstat

vmstat

provides a snapshot of processes, memory, paging, block I/O, traps, and CPU activity. Columns include runnable processes (r), uninterruptible sleep (b), swapped memory (swpd), buffers, cached, block I/O (bi/bo), interrupts (in), and context switches (cs).

Experiments with different

-j

values when compiling show that context‑switch rates remain stable until the parallelism level is pushed high enough to cause noticeable increases.

pidstat

pidstat

offers per‑process statistics, including page faults (

minflt/s

minor,

majflt/s

major), stack usage, CPU usage, and thread‑level context switches. Options such as

-t

(thread view),

-r

(memory),

-s

(stack),

-u

(CPU), and

-w

(context switches) make it ideal for deep analysis of individual or multithreaded programs.

Other CPU Tools

For per‑CPU inspection on SMP systems,

mpstat -P ALL 1

shows load distribution across cores. Filtering

top

by user (

top -u username

) or using

ps

with custom columns can isolate specific processes, and

ps axjf

visualizes process trees.

Disk I/O Monitoring

iotop

visualizes real‑time disk read/write rates per process.

lsof

reveals which processes hold files or devices open, useful for diagnosing unmount failures.

iostat -xz 1

reports key disk metrics: average queue length (

avgqu-sz

), average request latency (

await

), service time (

svctm

), and utilization (

%util

). Values >1 for

avgqu-sz

or >60% for

%util

indicate potential saturation.

These metrics also apply to network file systems, though kernel I/O caching can mask some performance impacts.

Network Monitoring

Network health is critical for servers.

iptraf

and

sar -n DEV 1

show interface throughput and utilization.

netstat

netstat -s

displays cumulative protocol statistics since boot;

netstat -antp

lists active TCP connections, while

netstat -nltp

shows listening sockets.

sar

Using

sar -n TCP,ETCP 1

provides per‑second TCP metrics such as active opens, passive opens, retransmissions, and input errors. For UDP,

sar -n UDP 1

reports packets received on closed ports and input errors, helping assess reliability.

tcpdump

tcpdump

captures raw packets for offline analysis with Wireshark. It supports size‑based rotation (

-C

/

-W

) and extensive filtering (interface, host, port, protocol). Captured packets include timestamps, enabling precise reconstruction of connection sequences, though the tool adds overhead that must be considered in production.