Master Linux System Performance: Top, Perf, Sar, Vmstat, and More

1. top

The top command displays real‑time system resource usage (CPU, memory, etc.). It reads load averages from /proc/loadavg (1‑, 5‑, and 15‑minute averages) and process counts from /proc/stat. Example to capture thread‑level data every second for 200 samples: top -H -b -d 1 -n 200 > top.txt CPU fields from /proc/stat: us – user time (nice ≤ 0) ni – nice time (nice > 0) sys – kernel time (excluding interrupts) id – idle time wa – I/O wait hi – hardware interrupts si – software interrupts st – steal time

2. perf

The Linux perf suite provides system‑wide profiling. sudo perf top -s comm shows the proportion of CPU time each process consumes (total time, not separated into idle/system/user). To record a profile and view it later:

sudo perf record

sudo perf report -s comm

3. sar and ksar

sar

(System Activity Report) collects real‑time and historical system activity. Install with sudo apt install sysstat and enable data collection by setting ENABLED="true" in /etc/default/sysstat. Dump all metrics to a file: LC_ALL=C sar -A > sar.txt Visualise the data with ksar (download from GitHub releases: https://github.com/vlsi/ksar/releases). Load the sar.txt file via Data → Load from text file… . ksar can plot CPU, memory, paging, block‑device I/O, filesystem, and swap statistics.

4. mpstat

mpstat

reports per‑CPU statistics. Example to sample every 10 seconds for 20 intervals on all CPUs: mpstat -P ALL 10 20 Typical fields: usr (user), nice, sys, iowait, irq, soft, idle, guest, gnice.

5. uptime

uptime

shows how long the system has been running and the 1‑, 5‑, 15‑minute load averages, reading data from /proc/uptime and /proc/loadavg.

11:15:41 up 82 days, 20:34, 8 users, load average: 0.28, 0.40, 0.43

6. vmstat

vmstat

focuses on memory usage but also reports CPU activity. Run vmstat 5 5 to collect five samples at five‑second intervals. Output columns are grouped into processes, memory, swap, I/O, system, and CPU sections.

procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st

7. pidstat

pidstat

monitors resource usage per process or thread.

# Show all processes, including idle
pidstat -p ALL
# Include thread statistics
pidstat -p ALL -t
# Specify interval and count
pidstat [options] interval [count]

Common options: -r for memory, -d for disk I/O, -u for CPU, etc. Output fields include page faults, virtual/real memory size, %MEM, and command name.

8. time

The time command measures CPU time taken by a program.

real    0m0.263s
user    0m0.094s
sys     0m0.011s

9. cpustat

Install with sudo apt install cpustat. It prints load average, CPU frequency, context switches, IRQs, and per‑process CPU usage.

Load Avg 0.66 0.54 0.49, Freq Avg. 1.46 GHz, 4 CPUs online

10. htop

htop

is an interactive, more readable version of top. Press F5 to toggle a tree view of threads and parent‑child relationships.

11. atop

atop

provides a top‑like view with additional columns for disk and network usage per process.

12. glances

glances

(Python‑based) reports CPU, memory, network, disk, and process statistics in a single screen. Press h for help.

13. nmon

nmon

displays CPU, memory, network, disk, and process information on a single screen and can export data to spreadsheets.

14. pcp‑gui

Performance Co‑Pilot (PCP) aggregates performance data from multiple hosts. Install with: sudo apt install pcp pcp-gui Open the GUI and select views (CPU, Disk, Memory, etc.) via the menu.

15. collectl and colplot

collectl

is a versatile command‑line tool that gathers a wide range of subsystem metrics (CPU, disk, memory, network, sockets, NFS, Lustre, etc.). Install with: sudo apt-get install collectl Basic usage displays CPU, disk, and network stats; collectl --all shows everything, and collectl --top replaces top.

collectl --all

colplot

visualises collectl data in a web browser. Download from SourceForge (https://sourceforge.net/projects/collectl-utils/files/), run sudo ./INSTALL, restart Apache, then open http://127.0.0.1/colplot/ to configure and generate plots.

sudo ./INSTALL
sudo systemctl restart apache2

References

/proc/stat: https://www.cnblogs.com/arnoldlu/p/9187775.html#system_proc_stat

/proc//stat: https://www.cnblogs.com/arnoldlu/p/9187775.html#process_proc_stat

System‑level performance analysis tool perf: https://www.cnblogs.com/arnoldlu/p/6241297.html

ksar: https://sourceforge.net/projects/ksar/

How to create sar graphs with ksar: https://www.cyberciti.biz/tips/identifying-linux-bottlenecks-sar-graphs-with-ksar.html

Collect and report Linux system activity information with sar: https://www.thomas-krenn.com/en/wiki/Collect_and_report_Linux_System_Activity_Information_with_sar

Linux performance measurements using vmstat: https://www.thomas-krenn.com/en/wiki/Linux_Performance_Measurements_using_vmstat

Collectl – the all‑round champion of Linux performance monitoring: https://linux.cn/article-3154-1.html

Collectl documentation: http://collectl.sourceforge.net/Documentation.html

Collectl examples: https://linoxide.com/monitoring-2/collectl-tool-install-examples/