Beyond top: 5 Linux Performance Commands That Save Production Systems

Problem Background

When production systems fail, 90% of junior ops engineers reach for top. But top only shows high-CPU processes and memory usage; it cannot answer why CPU is high, why I/O is high, why the network is saturated, or why the system is slow despite low CPU.

Real fault scenarios include:

Application latency spikes 10x while CPU <10% and I/O normal

Database slow but disk util <30%

OOM kill triggered despite free memory

Network jitter with low iftop traffic

RPS stalls despite stable process count

These require deeper tools: vmstat for system-wide metrics, mpstat / pidstat for per-CPU/process stats, iostat for disk I/O, sar for history, perf / bpftrace for kernel hotspots.

Five Layers of Performance Analysis

System-wide: vmstat, mpstat, sar Per-CPU: mpstat Per-process: pidstat, top, ps Disk I/O: iostat, iotop Network: sar, nicstat,

tcpdump

USE Method (Brendan Gregg)

For each resource (CPU, memory, disk, network) check:

Utilization: resource busy percentage

Saturation: queue length / wait time

Errors: error counters

Five Life-Saving Commands

vmstat

: overall system state (CPU, memory, I/O, system) mpstat: per-CPU statistics pidstat: per-process CPU, memory, I/O, context switches iostat: disk I/O statistics sar: historical performance data

Plus top as a supplement.

When to Use Which Command

Overall system state → vmstat Multi-core CPU balance → mpstat Which process consumes resources → pidstat Disk I/O bottleneck → iostat Historical performance data → sar Kernel hotspots →

perf

Key Performance Indicators

CPU

us

(user), sy (system), wa (iowait), id (idle), st (steal)

Memory

swpd

, free, buff/cache, si/so (swap in/out)

I/O

bi/bo

(blocks in/out), await (avg wait ms), %util (device utilization)

Process

cswch/s

(voluntary context switches), nvcswch/s (non-voluntary), minflt/s,

majflt/s

Overall Troubleshooting Flow

System layer: vmstat 1 for CPU/I/O/memory overview

CPU layer: mpstat -P ALL 1 for per-core load

Process layer: pidstat -u -r -d 1 for per-process details

Device layer: iostat -xz 1 for disk

History layer: sar for past data

Each step confirms or excludes a class of root causes.

Practical Diagnosis Scenarios

1. Application Slow, CPU Not High

Initial check: top -bn1 | head -20 + uptime. High load average (> cores) but low CPU% → likely I/O block.

Verify with vmstat: vmstat 1 10. Key columns: b (blocked processes), wa (iowait %), bi/bo. Example: b=8, wa=82% → 8 processes waiting on I/O, 82% CPU waiting.

Pinpoint disk: iostat -xz 1 5. Look for %util=100%, await=640ms, avgqu-sz=32 → disk saturated, likely heavy random writes.

Find offending process: pidstat -d 1.

Fixes: move database binlog to separate disk, add memory to turn reads into cache hits, optimize queries.

Verify: re-run vmstat; b and wa should drop.

2. Single CPU Core at 100%, Others Idle

Check: mpstat -P ALL 1 5 shows one core at 100% user, others 0%.

Identify process: pidstat -u -p <pid> 1 5 reveals single-threaded app (typical Python, PHP, old JVM).

Fixes: upgrade to multi-thread, run multiple processes behind load balancer, set CPU affinity to spread across cores.

3. High Memory Usage but Application Small

Check: free -h shows low available despite high buff/cache.

Process RSS: pidstat -r -p <pid> 1 5.

Real usage: smem -p -P myapp gives USS/PSS/RSS.

Root causes: application leak (RSS growing), kernel cache pressure, shared memory overcount.

Fixes: heap dump/jmap/pympler for leaks; drop_caches or add RAM for cache; reduce process count or use mmap for shared memory.

4. Disk I/O Jitter, Await Spikes

iostat -xz 1 10

: await > 10ms warning, >50ms critical; %util >80% near saturation. pidstat -d 1 5 and iotop -ao show which process writes.

Common causes: excessive logging, database binlog storms, concurrent backups.

Fixes: adjust log policy, move binlog to separate disk/compress, stagger backups.

5. Network Latency High, Bandwidth Not Full

sar -n DEV 1 5

shows low throughput. sar -n EDEV 1 5 reveals rxdrop=50/s → ring buffer full. ss -s and ss -tan for connection stats.

Root causes: ring buffer exhaustion, IRQ affinity imbalance, conntrack table full, TCP backlog full.

Fixes: increase ring buffer ( ethtool -G), multi-queue IRQ binding, raise net.core.somaxconn, reduce short connections.

6. Historical Analysis with sar

Use sar -f /var/log/sa/saDD with flags: -u CPU, -r memory, -b/-d I/O, -n DEV/EDEV/TCP network, -q load, -S/-W swap. Essential for post-mortem.

7. High Context Switches

vmstat 1 5

: cs > 100k/s high. pidstat -w 1 5 shows per-process cswch/s (voluntary) vs nvcswch/s (involuntary). High voluntary → lock contention, too many processes, short connections.

8. High Soft Interrupts

mpstat 1 5

: %soft high. mpstat -P ALL 1 5 identifies specific cores. Usually multi-queue NIC IRQ imbalance.

Fixes: set IRQBALANCE_BANNED_CPUS, disable RPS/RFS ( ethtool -K), upgrade NIC driver.

9. Kernel Hotspots with perf

perf top

for real-time. perf record -F 99 -a -g -- sleep 30 then perf report --stdio or perf script for call stacks.

10. Kernel Tracing with bpftrace

Install: yum install bpftrace. Example I/O latency histogram:

bpftrace -e 'kprobe:blk_mq_start_request { @start[arg0] = nsecs; } kprobe:blk_mq_end_request /@start[arg0]/ { @usecs = hist((nsecs - @start[arg0]) / 1000); delete(@start[arg0]); }'

. TCP retransmits: bpftrace -e 'kprobe:tcp_retransmit_skb { @retrans++; }'.

Common Command Reference

vmstat

vmstat 1          # every second
vmstat 1 10       # 10 samples
vmstat -s         # cumulative stats
vmstat -d         # disk stats
vmstat -p /dev/sda1 # specific partition

Ignore first line (average since boot).

mpstat

mpstat 1              # all CPUs average
mpstat -P ALL 1       # per core
mpstat -P 0 1         # CPU 0
mpstat -u 1           # CPU utilization

%iowait >30%

suggests I/O bottleneck; %soft high indicates network/block softirqs.

pidstat

pidstat 1           # all processes
pidstat -p 1234 1   # specific PID
pidstat -u 1        # CPU
pidstat -r 1        # memory
pidstat -d 1        # I/O
pidstat -w 1        # context switches
pidstat -t 1        # threads

Advantages over top: shows CPU affinity and per-process I/O rates ( kB_rd/s, kB_wr/s).

iostat

iostat -xz 1        # extended, per second
 iostat -d 1        # disk stats
 iostat -p sda 1    # specific disk
 iostat -m 1        # MB/s
 iostat -N 1        # LVM devices

%util >80%

sustained 5 minutes typically indicates I/O bottleneck.

sar

sar 1 5             # all basic metrics
sar -u 1 5          # CPU
sar -r 1 5          # memory
sar -b 1 5          # I/O
sar -d 1 5          # per device
sar -n DEV 1 5      # network traffic
sar -n EDEV 1 5     # network errors
sar -n TCP 1 5      # TCP
sar -q 1 5          # load
sar -S 1 5          # swap
sar -W 1 5          # swap in
sar -A              # all
sar -f /var/log/sa/sa25 # historical data

Data collector is sadc (sysstat package), default 10-min interval, 7-day retention.

top

top -bn1            # static output
 top -p 1234         # specific PID
 top -H              # threads
 top -c              # show command line
 top -o %CPU         # sort by CPU
 top -o %MEM         # sort by memory
 top -u myapp        # specific user

Good for quick glance, not for monitoring scripts.

Other Tools

htop

: enhanced top iotop: real-time disk I/O iftop: real-time network traffic nethogs: per-process network ss: socket statistics lsof: open files pstree: process tree perf: kernel profiling bpftrace: eBPF tracing smem: real memory usage (USS/PSS) numastat: NUMA memory

sysstat Installation & Configuration

yum install sysstat
systemctl enable sysstat
systemctl start sysstat

/etc/sysconfig/sysstat

:

HISTORY=28
COMPRESSAFTER=14
SADC_OPTIONS="-S DISK"

Retain 28 days, compress after 14.

Cron ( /etc/cron.d/sysstat):

*/10 * * * * root /usr/lib64/sa/sa1 1 1
53 23 * * * root /usr/lib64/sa/sa2 -A

Sample every 10 minutes, daily summary at 23:53.

Monitoring Integration

Prometheus node_exporter

scrape_configs:
- job_name: 'node'
  static_configs:
  - targets: ['node_exporter:9100']
  scrape_interval: 15s

Grafana Dashboard Variables

$instance: hostname
$cpu: cpu number
$device: disk device
$nic: network interface

Alerting Rules (Prometheus)

groups:
- name: cpu
  rules:
  - alert: HighCpuLoad
    expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
    for: 10m
    annotations:
      summary: "CPU > 80% on {{ $labels.instance }}"

Thresholds must align with business baselines, not hardcoded.

Threshold Guidelines (Adjust to Baseline)

CPU us: Warning >70%, Critical >90%

CPU wa: Warning >20%, Critical >40%

load avg: Warning > cores, Critical > 2x cores

Memory available: Warning <20%, Critical <10%

swap si/so: Warning >0, Critical >1000/s

iostat await: Warning >10ms, Critical >50ms

iostat %util: Warning >70%, Critical >90%

nic rxdrop: Warning >10/s, Critical >100/s

Context switches: Warning >50k/s, Critical >200k/s

%soft: Warning >10%, Critical >30%

Collection Intervals

Real-time troubleshooting: 1 second

Daily monitoring: 10 seconds

Historical archiving: 10 minutes

Long-term trends: 1 hour

Correlation Analysis

High CPU + high wa = I/O bottleneck

High CPU + high sy = kernel bottleneck (locks, syscalls)

High CPU + high us = application bottleneck

High memory + swap >0 = memory pressure

High await + high %util = disk saturation

High await + low %util = queueing issue (NVMe multi-queue, SSD controller)

Troubleshooting Paths

"Application Slow"

top

for CPU hogs vmstat for wa mpstat for single-core saturation pidstat for per-process resources iostat for disk sar -n DEV for network

Application slow-query logs

"Database Slow"

Slow query log

SHOW PROCESSLIST

SHOW ENGINE INNODB STATUS

pidstat -d

for disk I/O iostat -xz for await EXPLAIN for execution plan

"Network Slow"

sar -n DEV/EDEV

ping

baseline latency traceroute path ss -s socket stats tcpdump capture ethtool NIC stats

"Memory Slow"

free -h

vmstat

for swap pidstat -r for process RSS /proc/meminfo detail slabtop kernel slab drop_caches to release

"Disk Slow"

iostat -xz

iotop

real-time pidstat -d process I/O dmesg for I/O errors smartctl disk health fio performance test

Risk Warnings

kill -9 <pid>

: may lose data; try SIGTERM first echo 3 > /proc/sys/vm/drop_caches: only frees buff/cache, use cautiously in prod sysctl -p: immediate effect; wrong change can break network perf record -a -g: high overhead sampling, may slow system bpftrace: requires kernel ≥4.9, non-negligible overhead dd if=/dev/zero of=/data/test bs=1G count=10: fills disk stress --cpu 8: may trigger other service circuit breakers kill -STOP <pid>: pauses process, may be misdetected as hang tc qdisc add dev eth0 root netem delay 1000ms: network emulation must avoid production impact iperf3 -c server: bandwidth test may be flagged by monitoring smartctl --smart-test: short test safe, long test needs window hdparm -Tt /dev/sda: may affect business I/O fio --rw=randwrite: high I/O test risks disk wear tcpdump -i any: may capture passwords; ensure compliance perf stat -e cache-misses: long sampling has high overhead sysctl vm.swappiness=0: may cause OOM chmod -R 777 /proc/sys/vm: breaks sysctl echo c > /proc/sysrq-trigger: forces crash; debug only

Verification Checklist

Key metrics match expected values

Business latency restored

No new alerts introduced

Resource utilization back to baseline

Error logs zero

Application metrics normal

Historical trend analysis: sar 7-day comparison

Regression test: simulate production traffic

Monitoring dashboards show healthy trends

Capacity planning: any emerging bottlenecks

Rollback Procedures

sysctl Rollback

# 1. Backup
cp /etc/sysctl.d/99-custom.conf /etc/sysctl.d/99-custom.conf.bak
# 2. Restore original values
vi /etc/sysctl.d/99-custom.conf
# 3. Reload
sysctl -p /etc/sysctl.d/99-custom.conf

Process Rollback

systemctl stop myapp
ln -sfn /opt/myapp-1.0 /opt/myapp-current
systemctl start myapp

Database Parameter Rollback

SET GLOBAL innodb_buffer_pool_size = 4*1024*1024*1024;

Or restart with old my.cnf.

Kernel Parameter Rollback

sysctl net.core.somaxconn=128

Config File Rollback

cp /etc/my.cnf.bak /etc/my.cnf
systemctl restart mysqld

Disk I/O Scheduler Rollback

echo cfq > /sys/block/sda/queue/scheduler

Network Parameter Rollback

tc qdisc del dev eth0 root
ethtool -G eth0 rx 256 tx 256

Service Restart Rollback

cp /etc/myapp/myapp.conf.bak /etc/myapp/myapp.conf
systemctl restart myapp

Kernel Upgrade Rollback

dnf remove kernel-5.14.0-new
grub2-mkconfig -o /boot/grub2/grub.cfg
reboot

Full Rollback Checklist

Config backups

Binary backups

Database backups

Change records

Monitoring metric trends

Business impact assessment

Canary strategy

Production Environment Guidelines

Performance tuning during low-traffic windows

Canary: validate on one node first

Backup before every change

All changes monitored

Document every change

Pre-change drills

Continuous data collection during tuning

Before/after metric comparison mandatory

Every change must be reversible

Assess blast radius

Capacity planning considered

Business stakeholder sign-off

Team notification

Alert thresholds adjusted during change

Thorough performance testing

Summary

top

is entry-level; it alone cannot save production. The five commands each have a role: vmstat: quick system-wide view mpstat: multi-core balance issues pidstat: pinpoint specific process iostat: disk bottleneck location sar: historical retrospection

Add perf and bpftrace to cover 90% of performance issues.

Troubleshooting mindset: system before process, CPU before I/O, real-time before history.

Remember: performance problems are never single-point; correlate multiple metrics, don't stare at one number.

Master these five commands; practice beats reading 100 tutorials.

12 Rules of Thumb for Performance Analysis

System before individual : vmstat / mpstat first

Real-time before history : vmstat 1 vs sar -f CPU before I/O : iowait is key intermediate

Resource before application : only if system resources OK look at app

Process before thread : pidstat -t for threads

Kernel before user : perf top for kernel

Frequency before bandwidth : perf stat for cache misses

Count before latency : sar -q vs sar -n Trend before peak : 7-day trend vs 5-min peak

Comparison before absolute : cross-host vs single-host

Baseline before anomaly : establish baseline to know deviation

Slow before fast : prioritize slowest metrics

Performance Tuning Cheat Sheet

CPU Scheduling: nice, taskset → commands: nice, taskset

CPU Affinity: pinning → commands: numactl, taskset

Filesystem: mount options → command: mount -o noatime

I/O Scheduler: choose bfq/none → command: echo bfq > scheduler

Memory Reclaim: swappiness → command: sysctl vm.swappiness

Network Stack: tcp params → command: sysctl net.ipv4.tcp_*

Process Limits: ulimit → command: ulimit -n

Open Files: file-max → command: sysctl fs.file-max

Tool Collaboration

vmstat

+ dstat: dstat -tldr (enhanced vmstat) iostat + iotop: iotop real-time, iostat average mpstat + perf: mpstat finds busy core, perf profiles that core ( perf top -C 0) pidstat + pstree: pidstat identifies process, pstree shows hierarchy sar + Grafana: sar collects, Grafana visualizes perf + bpftrace: perf for sampling, bpftrace for targeted tracing

Monitoring Design Recommendations

Layers

L1 System: CPU, memory, I/O, network

L2 Process: per-process CPU, memory, I/O

L3 Application: QPS, latency, error rate

L4 Business: orders, users

Collection Frequency

L1: 10s

L2: 30s

L3: 60s

L4: 5min

Retention

Real-time: 1 hour

Medium: 7 days

Long-term: 1 year

Alert Tiers

Critical: business impact

Warning: resource pressure

Info: baseline deviation

Alert convergence: correlate multiple alerts to single root cause.

Avoid alert fatigue: thresholds based on baselines, not hardcoded.

Monthly alert drills.

Performance Baselines

Host Baseline

CPU avg utilization <30%

load avg < core count

Memory available >30%

Swap in = 0

Disk %util <50%

Network bandwidth <30%

Database Baseline

QPS <50% of max

Slow queries <10/min

Replication lag <1s

Connections <50% max

Application Baseline

P99 latency <50% of baseline

Error rate <0.1%

Process count stable

Container Performance Troubleshooting

Container vmstat

docker exec <container> vmstat 1

Container cgroup CPU

cat /sys/fs/cgroup/cpu/system.slice/docker-<id>.scope/cpuacct.usage

Container I/O

cat /sys/fs/cgroup/blkio/system.slice/docker-<id>.scope/blkio.throttle.io_service_bytes

Container perf

Requires --cap-add=SYS_PTRACE or privileged mode.

Kubernetes

kubectl top pod <pod>
kubectl top node <node>
kubectl describe pod <pod>

Depends on metrics-server.

Stress Testing Tools

fio : disk I/O stress

iperf3 : network stress

sysbench : CPU/memory/database

wrk : HTTP stress

ab : simple HTTP stress

stress-ng : general stress

netperf : network performance

dd : simple disk read

Production Stress Testing Rules

Must run in isolated environment

Never stress production

Avoid impacting live users

Must have monitoring

Must have owner

Must have rollback plan

Must have SLA monitoring

Case Study: E-commerce Peak Optimization

Background

Pre-sale: application slow, expecting 10x traffic.

Diagnosis

vmstat

: high wa mpstat: CPU 0 saturated pidstat: myapp process iostat: sda saturated perf top: kernel mutex_lock hotspot

Root Causes

Single-threaded application

Database random I/O heavy

Kernel lock contention

Optimizations

Rewrite app to multi-threaded

Add database indexes

Lock-free critical paths

Add SSD cache disk

Validation

Stress test meets target QPS

Business P99 <50ms

Resource utilization <70%

Retrospective

Optimize holistically

Don't fixate on one metric

Stress test must mimic real workload

Optimization is iterative

Monitoring & Alerting Integration

Key Metric Alerts

- alert: HighCpuIowait
  expr: avg by(instance) (rate(node_cpu_seconds_total{mode="iowait"}[5m])) * 100 > 20
  for: 10m
  annotations:
    summary: "CPU iowait > 20% on {{ $labels.instance }}"

- alert: HighDiskUtil
  expr: 100 - (avg by(instance) (rate(node_filesystem_free_bytes{fstype!~"tmpfs|overlay"}[5m])) * 100) > 80
  for: 10m

Composite Signals

High CPU + High I/O = disk bottleneck

High CPU + High Memory = app bottleneck

High Network + High CPU = app bottleneck

Low CPU + High Latency = lock contention

Latency-Based Alerts

Business metrics more sensitive than resource metrics:

- alert: HighResponseTime
  expr: histogram_quantile(0.99, sum by(le) (rate(http_request_duration_seconds_bucket[5m]))) > 1
  for: 5m

Quick-Reference Mnemonics

top → overview

vmstat → system-wide

mpstat → per-core

pidstat → per-process

iostat → disk

sar → history

perf → hotspots

Run in order: system → process, CPU → I/O, real-time → history.

Recommended Reading

Brendan Gregg Systems Performance

Brendan Gregg Blog: http://www.brendangregg.com/

Linux Performance: http://www.brendangregg.com/linuxperf.html

USE Method (Brendan Gregg)

RED Method (Tom Wilkie)

Google SRE Book Chapter 11

Performance Analysis 2nd Edition

Final Thoughts

No silver bullet in performance troubleshooting—only methodology combined with tools. Master these five commands, follow the chapters when issues arise, and apply business context for judgment. This beats ad-hoc googling.

Performance optimization is continuous, not one-off. Establish baselines, monitor continuously, review regularly—three keys to sustainable performance.