Essential Linux Performance Tools: Quick Guide to Diagnose System Bottlenecks

Linux Performance Tools Overview

This guide is based on Netflix's "Linux Performance Analysis in 60,000 Milliseconds" blog and adds personal insights for quickly diagnosing system performance problems.

1. uptime

$ uptime
23:51:26 up 21:31, 1 user, load average: 30.02, 26.43, 19.02

The command shows overall system load; the three numbers are the average load over the past 1, 5, and 15 minutes.

2. dmesg | tail

$ dmesg | tail
[1880957.563150] perl invoked oom‑killer: gfp_mask=0x280da, order=0, oom_score_adj=0
[1880957.563400] Out of memory: Kill process 18694 (perl) score 246 or sacrifice child
[1880957.563408] Killed process 18694 (perl) total‑vm:1972392kB, anon‑rss:1953348kB, file‑rss:0kB
[2320864.954447] TCP: Possible SYN flooding on port 7001. Dropping request. Check SNMP counters.123456

Shows the kernel ring buffer, useful for spotting OOM kills, TCP errors, and other anomalies.

3. vmstat 1

$ vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
34  0      0 200889792  73708 591828    0    0     0     5    6   10 96  1  3  0  0
...

Displays statistics for processes, memory, swap, I/O, and CPU. Important columns to watch:

r – runnable or waiting processes (CPU saturation if > CPU cores)

free – free memory

si, so – swap in/out (non‑zero indicates memory pressure)

us, sy, id, wa – user, system, idle, and I/O wait CPU percentages (system time >20% may signal I/O bottleneck)

4. mpstat -P ALL 1

$ mpstat -P ALL
Linux 3.10.0-229.el7.x86_64 (localhost.localdomain) 05/30/2018 _x86_64_ (16 CPU)
04:03:55 PM  CPU   %usr   %nice   %sys %iowait %irq %soft %steal %guest %gnice %idle
04:03:55 PM  all    3.67    0.00    0.61   0.71   0.00   0.00   0.00   0.00   0.00  95.02
...

Shows per‑CPU utilization; useful for checking whether CPU scheduling is balanced across cores.

5. pidstat 1

$ pidstat 1
Linux 3.13.0-49-generic (titanclusters-xxxxx) 07/14/2015 _x86_64_ (32 CPU)
07:41:02 PM   UID   PID   %usr %system %guest %CPU  CPU  Command
07:41:03 PM    0     9   0.00   0.94   0.00  0.94   1   rcuos/0
07:41:03 PM    0   4214  5.66   5.66   0.00 11.32  15   mesos‑slave
07:41:03 PM    0   6521 1596.23 1.89   0.00 1598.11 27   java
...

Provides per‑process CPU usage; unlike top it scrolls continuously without clearing the screen. High %CPU values (e.g., >100% on multi‑core systems) indicate heavy CPU consumption.

6. iostat -xz 1

$ iostat -xz 1
Linux 3.13.0-49-generic (titanclusters-xxxxx) 07/14/2015 _x86_64_ (32 CPU)
avg-cpu:  %user   %nice %system %iowait %steal %idle
           73.96    0.00    3.73    0.03    0.06   22.21
Device: rrqm/s wrqm/s   r/s   w/s  rkB/s  wkB/s avgrq‑sz avgqu‑sz await r_await w_await svctm %util
xvda    0.00   0.23   0.00  0.21   0.18   4.52   2.08   34.37   0.00   0.00   0.00  0.09
...

First line is from boot to the moment of execution; subsequent lines are per‑interval statistics. Key columns:

r/s, w/s – reads/writes per second

await – average I/O wait time (high values indicate saturation)

avgqu‑sz – average queue length (value >1 suggests I/O overload)

%util – device utilization ( >60% often signals a problem)

7. free -m

$ free -m
              total   used   free  shared  buffers  cached
Mem:        245998  24545 221453     83      59    541
-/+ buffers/cache: 23944 222053
Swap:            0      0      0

Important fields:

buffers – block device cache

cached – page cache (used by file systems)

If buffers and cached are near zero, I/O usage is extremely high.

8. sar -n DEV 1

$ sar -n DEV 1
Linux 3.10.0-229.el7.x86_64 (localhost.localdomain) 05/31/2018 _x86_64_ (16 CPU)
03:54:57 PM IFACE   rxpck/s txpck/s  rxkB/s  txkB/s  rxcmp/s txcmp/s rxmcst/s
03:54:58 PM ens32   3286.00 7207.00 283.34 18333.90   0.00    0.00    0.00
03:54:58 PM lo        0.00    0.00   0.00    0.00   0.00    0.00    0.00
...

Columns:

IFACE – network interface name

rxpck/s, txpck/s – packets received/transmitted per second

rxkB/s, txkB/s – kilobytes received/transmitted per second

rxcmp/s, txcmp/s – compressed packets per second

rxmcst/s – multicast packets per second

9. sar -n TCP,ETCP 1

$ sar -n TCP,ETCP 1
04:16:27 PM active/s passive/s iseg/s oseg/s
04:16:44 PM   0.00   2.00   15.00   13.00
04:16:45 PM   0.00   3.00  126.00  203.00
...

Key columns:

active/s – locally‑initiated TCP connections per second

passive/s – remotely‑initiated TCP connections per second

retrans/s – TCP retransmissions per second

10. top

$ top
top - 00:15:40 up 21:56, 1 user, load average: 31.09, 29.87, 29.92
Tasks: 871 total, 1 running, 868 sleeping, 0 stopped, 2 zombie
%Cpu(s): 96.8 us, 0.4 sy, 2.7 id, 0.1 wa
KiB Mem: 25190241+ total, 24921688 used, 22698073+ free, 60448 buffers
...

top

aggregates many metrics into a single view, providing a comprehensive snapshot of system health.

Summary Diagram

The following image maps each command to the aspect of system performance it helps to monitor.

References

Performance not good? Follow the checklist.

Linux Performance Analysis in 60,000 Milliseconds (Netflix blog).

Netflix performance testing tools video tutorial.