Diagnose Linux Performance Issues in 1 Minute with 10 Essential Commands

Overview

If your Linux server suddenly experiences a massive load increase and alerts flood your phone, you can identify the performance bottleneck in under a minute using a concise set of diagnostic commands popularized by Brendan Gregg’s Netflix performance engineering team.

One‑Minute Health‑Check Commands

The following commands give a rapid snapshot of system resource usage:

uptime

dmesg | tail

vmstat 1

mpstat -P ALL 1

pidstat 1

iostat -xz 1

free -m

sar -n DEV 1

sar -n TCP,ETCP 1

top

Most of these utilities are provided by the sysstat package, while others come from procps. Their output follows the USE method (Utilization, Saturation, Errors) to quickly locate performance bottlenecks.

uptime

$ uptime

23:51:26 up 21:31, 1 user, load average: 30.02, 26.43, 19.02

This shows the system’s load averages for the past 1, 5, and 15 minutes. A high 1‑minute load with a much lower 15‑minute load indicates a recent spike that needs further investigation, whereas the opposite suggests the load spike has already subsided.

dmesg | tail

$ dmesg | tail

[1880957.563150] perl invoked oom‑killer: gfp_mask=0x280da, order=0, oom_score_adj=0
[1880957.563400] Out of memory: Kill process 18694 (perl) score 246 or sacrifice child
[1880957.563408] Killed process 18694 (perl) total‑vm:1972392kB, anon‑rss:1953348kB, file‑rss:0kB
[2320864.954447] TCP: Possible SYN flooding on port 7001. Dropping request.  Check SNMP counters.

The last ten kernel messages can reveal out‑of‑memory kills, network issues, or other kernel‑level events that often explain sudden performance degradation.

vmstat 1

$ vmstat 1

procs ---------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r  b swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
34  0    0 200889792  73708 591828    0    0     0     5    6   10 96  1  3  0  0
...

Key columns include:

r : processes waiting for CPU (more than CPU cores → CPU saturation)

free : available memory in kilobytes

si/so : swap in/out activity (non‑zero indicates memory pressure)

us, sy, id, wa, st : CPU time spent in user, system, idle, I/O wait, and stolen (hypervisor) states

High wa suggests I/O bottlenecks, while a large sum of us and sy indicates a busy CPU.

mpstat -P ALL 1

$ mpstat -P ALL 1

Linux 3.13.0-49-generic (titanclusters-xxxxx) 07/14/2015 _x86_64_ (32 CPU)
07:38:49 PM  CPU   %usr  %nice  %sys %iowait  %irq  %soft  %steal  %guest  %gnice  %idle
07:38:50 PM  all  98.47   0.00   0.75   0.00   0.00   0.00    0.00    0.00    0.00   0.78
07:38:50 PM    0  96.04   0.00   2.97   0.00   0.00   0.00    0.00    0.00    0.00   0.99
...

This displays per‑CPU utilization. A single CPU with a very high %usr or %sys often points to a single‑threaded workload dominating the system.

pidstat 1

$ pidstat 1

Linux 3.13.0-49-generic (titanclusters-xxxxx) 07/14/2015 _x86_64_ (32 CPU)
07:41:02 PM   UID   PID   %usr %system %guest %CPU CPU Command
07:41:03 PM    0    9   0.00   0.94   0.00  0.94  1 rcuos/0
07:41:03 PM    0 4214   5.66   5.66   0.00 11.32 15 mesos‑slave
07:41:03 PM    0 4354   0.94   0.94   0.00  1.89  8 java
07:41:03 PM    0 6521 1596.23  1.89   0.00 1598.11 27 java
07:41:03 PM    0 6564 1571.70  7.55   0.00 1579.25 28 java
...

The output shows each process’s CPU consumption. In the example, two Java processes together consume roughly 1600 % CPU, meaning they are using about 16 CPU cores, a clear indication of a severe CPU‑bound issue.

Other Useful Commands

While the article lists iostat -xz 1, free -m, sar -n DEV 1, sar -n TCP,ETCP 1, and top, they follow the same principle: provide real‑time metrics for disk I/O, memory availability, network statistics, and an interactive view of processes, respectively.

By running these ten commands, you obtain a comprehensive, high‑level picture of CPU, memory, I/O, and kernel health, enabling rapid identification of the root cause of a performance incident.