Master Linux Server Monitoring: Essential Tools & Metrics Explained

Running a Linux server generates a wealth of parameter data that is crucial for both operations staff and developers, especially when programs misbehave.

Below are simple tools for viewing system parameters; many of them analyze data from

/proc

and

/sys

, while more advanced monitoring may require tools like

perf

or

systemtap

.

1. CPU and Memory

1.1 top

➜ ~ top

The first line shows the 1‑, 5‑, and 15‑minute average load; values exceeding the number of CPU cores indicate saturation.

The second line lists task states: running , sleeping , stopped , and zombie , each with specific meanings.

The third line breaks down CPU usage:

us (user): time spent in user mode with low nice values.

sy (system): time spent in kernel mode, often higher during heavy I/O.

ni (nice): time spent in user mode with positive nice values.

id (idle): idle time.

wa (iowait): time waiting for I/O completion.

hi (irq): time handling hardware interrupts.

si (softirq): time handling software interrupts.

st (steal): time lost to the hypervisor in virtualized environments.

High CPU usage hints at specific issues:

If us is high, a user process is consuming CPU; locate it with

top

or

perf

.

If sy is high, heavy I/O or kernel problems may be the cause.

If ni is high, the process was intentionally nicened.

If wa is high, I/O efficiency is low.

If irq/softirq is high, hardware may be misbehaving.

If st is high, the VM is likely oversold.

The fourth and fifth lines show physical and virtual memory details.

total = free + used + buff/cache

;

Buffers

cache raw disk metadata, while

Cached

caches file data.

Avail Mem

indicates memory available without swapping.

Note that

top

itself consumes resources and is best for real‑time, short‑term monitoring.

1.2 vmstat

vmstat

provides another view of system load.

Key fields: r (runnable processes), b (uninterruptible sleep), swpd (used swap), bi/bo (blocks I/O), in (interrupts per second), cs (context switches per second).

When compiling with

-j

, context switches only increase noticeably after a high

-j

value, suggesting the parameter need not be overly aggressive.

1.3 pidstat

pidstat -t -C "ailaw" -l

Provides per‑process statistics, including:

-r : page faults (minor

minflt/s

and major

majflt/s

).

-s : stack usage (

StkSize

and

StkRef

).

-u : CPU usage.

-w : thread context switches (

cswch/s

and

nvcswch/s

).

Using

-C

filters by command name, making pidstat more convenient than

ps

for multithreaded processes.

1.4 Other CPU tools

For per‑CPU monitoring,

mpstat -P ALL 1

shows load distribution across cores.

To filter processes by user:

top -u taozj

or custom

ps

formats, e.g.:

while :; do ps -eo user,pid,ni,pri,pcpu,psr,comm | grep 'ailawd'; sleep 1; done

Process tree can be displayed with

ps axjf

.

2. Disk I/O

iotop

visualizes per‑process disk read/write rates;

lsof

shows which processes have files open, useful for diagnosing unmount issues.

2.1 iostat

iostat -xz 1

Key metrics:

avgqu-s : average queue length; >1 indicates saturation for a single disk.

await (r_await, w_await): average I/O request latency.

svctm : average service time; close to await means low wait time.

%util : device utilization; >60% degrades performance, near 100% means saturation.

These metrics also apply to network file systems.

3. Network

Network performance is critical; tools like

iptraf

,

sar -n DEV 1

, and

netstat

help monitor throughput, packet loss, and retransmissions.

3.1 netstat

netstat -s

shows protocol statistics since boot; useful for checking ports and connections with options such as

-antp

(TCP) and

-nltp

(listening sockets).

3.2 sar

sar -n TCP,ETCP 1

reports TCP activity, including active/s , passive/s , retrans/s , and isegerr/s . For UDP,

sar -n UDP 1

shows noport/s and idgmerr/s , indicating packets without listeners or other errors.

3.3 tcpdump

tcpdump

captures packets for offline analysis with Wireshark; use

-C

/

-W

to limit file size and rotate captures.