Master Linux System Monitoring: Top, Perf, Sar, Vmstat, and More

01. top

top is the most common tool to view system resource usage, focusing on CPU.

/proc/loadavg

Load average is read from /proc/loadavg. Example output: 9.53 9.12 8.37 3/889 28165 The first three numbers are the 1‑, 5‑, and 15‑minute average of runnable and running processes. The fourth field shows running processes over total processes, and the last field is the most recent PID. top uses the first three numbers.

Using top

Run top with options to set the update interval, show threads, or display a single‑CPU view, and redirect output to a file. top -H -b -d 1 -n 200 > top.txt top relies on /proc/stat and /proc//stat for its data.

CPU fields from /proc/stat: us: user time (nice ≤ 0) ni: nice time (nice > 0) sys: system time (kernel, no interrupts) id: idle time wa: I/O wait hi: hardware interrupts si: software interrupts st: steal time

02. perf

perf is a system‑level performance analysis tool. Use sudo perf top -s comm to see per‑process CPU share, and sudo perf record followed by sudo perf report -s comm for detailed sampling.

03. sar and ksar

sar (System Activity Report) records real‑time system activity; ksar visualises sar data.

Install sar with: sudo apt install sysstat Enable it by editing /etc/default/sysstat (set ENABLED="true") and adjust the schedule in /etc/cron.d/sysstat. Restart the service:

# Enable sar
sudo gedit /etc/default/sysstat
# Edit cron schedule
sudo gedit /etc/cron.d/sysstat
# Restart service
sudo /etc/init.d/sysstat restart
# List logs
ls -l /var/log/sysstat/

Collect data (all metrics) to a file: LC_ALL=C sar -A > sar.txt Load the file in ksar via Data → Load from text file…

04. mpstat

mpstat shows per‑CPU statistics. Example usage:

mpstat -P ALL 10 20

Columns: usr (user), nice, sys (kernel), iowait, irq, softirq, idle, guest, gnice.

05. uptime

uptime reports total system up time and load averages from /proc/uptime and /proc/loadavg.

11:15:41 up 82 days, 20:34, 8 users, load average: 0.28, 0.40, 0.43

06. vmstat

vmstat monitors memory usage and also includes CPU information.

Example command vmstat 5 5 produces:

procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0 472576 228688 559092 1061756    0    0     9    39    1    0  8  4 87  0  0

The output is divided into six groups: processes, memory, swap, I/O, system, and CPU.

07. pidstat

pidstat monitors resource usage per process.

# Show all processes including idle
pidstat -p ALL
# Show thread details
pidstat -p ALL -t
# Interval sampling
pidstat [option] interval [count]

Use -r for memory, -d for disk I/O.

15时18分21秒   UID PID minflt/s majflt/s VSZ RSS %MEM Command
0 0 1 0.02 0.00 185316 3028 0.08 systemd

Field meanings: minflt/s (minor page faults), majflt/s (major page faults), VSZ (virtual memory in kB), RSS (resident set size), %MEM (memory percentage), Command (process command).

08. time

The time command measures CPU time of a program.

real 0m0.263s
user 0m0.094s
sys 0m0.011s

Subtracting user and sys from real gives I/O wait.

09. cpustat

Install cpustat and run cpustat -T -D -x to obtain load average, CPU frequency, context switches, IRQs, per‑task CPU usage, etc.

Load Avg 0.66 0.54 0.49, Freq Avg. 1.46 GHz, 4 CPUs online
3791.1 Ctxt/s, 1709.9 IRQ/s, 1800.0 softIRQ/s, 0.0 new tasks/s, 1 running, 0 blocked
%CPU %USR %SYS PID S CPU Time Task
25.74 25.74 0.00 11435 R 3 2.29w /usr/bin/python3
...

10. htop

htop provides a more readable interface than top, with thread view (F5) and tree view for parent‑child relationships.

11. atop

atop monitors system resources and processes, sorting by CPU usage and showing CPU, memory, disk and network per process.

12. glances

glances, written in Python, reports CPU, memory, network, disk and process statistics. Press “h” for help.

13. nmon

nmon displays CPU, memory, network, disk usage and process list on a single screen and can export data to spreadsheets.

13. pcp‑gui

Performance Co‑Pilot (PCP) collects and analyses performance data from multiple hosts. Install with: sudo apt install pcp pcp-gui Open views (CPU, Disk, Memory, etc.) via File → Open View.

14. collectl and colplot

collectl is a versatile command‑line tool that gathers performance data for CPU, disk, memory, network, sockets, NFS, Lustre, and many other subsystems, effectively replacing tools such as top, vmstat, ps, iotop, etc. sudo apt-get install collectl Running collectl without options shows CPU, disk and network. Use uppercase options for detailed per‑device statistics (e.g., C for per‑CPU, D for per‑disk).

Collect all subsystem data with collectl --all and view in a browser using colplot.

colplot visualises collectl data. Install from source, reload Apache, then access http://127.0.0.1/colplot/ to generate plots.