How to Pinpoint Bottlenecks on a Slowing Linux Server with top, vmstat, and iostat

Problem Overview

In production a Linux server may start fast but become sluggish after a period of uptime, causing slow page loads, request time‑outs and even SSH latency. The slowdown can stem from CPU, memory, disk I/O or network issues.

Typical Bottleneck Categories

CPU bottleneck : high user‑mode usage, many runnable processes, low idle time.

Memory bottleneck : low available memory, heavy swap activity, memory leaks, OOM killer.

Disk I/O bottleneck : high read/write IOPS, saturated bandwidth, long latency, random I/O.

Network bottleneck : bandwidth saturation, high latency, packet loss.

Using top to Locate CPU & Memory Issues

Run top and sort by CPU ( P) or memory ( M). Key fields: load average: 1‑, 5‑, 15‑minute averages. %Cpu(s): us (user), sy (system), id (idle), wa (iowait).

Process list shows %CPU, %MEM, PID, COMMAND.

Typical judgments:

If id < 10% and us > 80% → CPU bound.

If wa > 10% → I/O wait, investigate disk.

If a single process shows %CPU > 100% (multithreaded) → possible infinite loop or inefficient algorithm.

If memory usage is high and available is low → memory pressure.

Using vmstat for System‑Level Metrics

Run vmstat 2 10 to get a snapshot every 2 seconds. Important columns: r: runnable processes (waiting for CPU). b: processes blocked in uninterruptible I/O wait. swpd, free, buff, cache: memory usage. si / so: swap in/out rates. bi / bo: block I/O (blocks/s). us, sy, id, wa: CPU percentages. cs: context switches per second.

Interpretation examples: r > CPU cores → CPU queue overload. si/so > 0 with low free → memory shortage causing swap. b > 0 and high wa → many processes blocked on I/O.

Using iostat to Diagnose Disk Performance

Install sysstat if needed and run:

# yum install sysstat   # CentOS
# apt install sysstat   # Ubuntu

# Show extended stats every 2 s, 5 samples
iostat -x 2 5

Key fields per device: r/s, w/s: IOPS. rkB/s, wkB/s: throughput. r_await, w_await: average latency (ms). %util: percentage of time the device was busy (≈100% means saturation). aqu-sz: average queue length.

Typical judgments: %util > 90% → disk saturated.

High r_await / w_await (>10 ms) → latency problem, often mechanical HDD.

Low rareq‑sz / wareq‑sz (<10 KB) → random I/O, consider SSD.

Diagnostic Workflow (Quick Checklist)

Run top – decide whether the issue is CPU, memory or I/O (check id, wa, available).

If I/O suspected, run vmstat – verify b, wa, swap activity.

Use iostat – pinpoint the slow device and see if it is IOPS‑limited or bandwidth‑limited.

Run top -p <PID> -H or iotop -o to find the offending process.

Inspect application logs, strace, lsof for root cause.

Apply targeted fixes (code changes, configuration tweaks, hardware upgrades).

Example Cases

Case 1 – CPU bottleneck : top shows us=95%, a Java process consumes 780% CPU (10 threads). top -H and strace reveal a busy loop. Fix by optimizing the algorithm or adding CPU cores.

Case 2 – Memory bottleneck : top reports available=200 MB and heavy swap. vmstat shows si=5000, so=3000. A Java process holds 18 GB RAM – likely a memory leak. Fix by fixing the leak or adding RAM.

Case 3 – Disk I/O bottleneck : High wa in top, vmstat shows many blocked processes, iostat reports %util=98% on sda with r_await=25 ms. iotop points to MySQL heavy reads. Remedy with indexes, larger cache or moving to SSD.

Optimization Recommendations

CPU : profile code, reduce lock contention, use nice/renice, bind processes with taskset, consider more cores.

Memory : fix leaks, tune vm.swappiness, increase RAM, enable huge pages.

Disk I/O : switch to SSD/NVMe, adjust I/O scheduler ( deadline / noop), enable write‑back cache, use RAID, add application‑level caching.

Monitoring & Alerting (Prometheus Rules)

# High CPU usage
- alert: HighCPUUsage
  expr: 100 - (avg(irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "CPU usage exceeds 80%"

# High I/O wait
- alert: HighIOWait
  expr: avg(irate(node_cpu_seconds_total{mode="iowait"}[5m])) * 100 > 20
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "I/O wait exceeds 20%"

# High memory usage
- alert: HighMemoryUsage
  expr: (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 > 90
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Memory usage exceeds 90%"

# High swap usage
- alert: HighSwapUsage
  expr: (1 - node_memory_SwapFree_bytes / node_memory_SwapTotal_bytes) * 100 > 50
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "Swap usage exceeds 50%"

# High disk utilization
- alert: HighDiskUtil
  expr: irate(node_disk_io_time_seconds_total[5m]) * 100 > 90
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Disk utilization exceeds 90%"

Common Pitfalls

Assuming high load average always means CPU overload – check wa to see if I/O is the cause.

Treating low free memory as shortage – look at available which includes reclaimable cache.

Seeing any swap usage as a problem – only alarm when si / so are non‑zero and sustained.

Interpreting %util=100% as disk failure on NVMe – also examine await and queue depth.

Key Metrics Summary

CPU bottleneck : top – id < 10%, us > 80%; vmstat – r > CPU cores.

Memory bottleneck : top – available < 10%, swap used > 0; vmstat – si/so > 0.

Disk I/O bottleneck : top – wa > 10%; vmstat – b > 0, high bi/bo; iostat – %util > 90%, await > 10 ms.

Final Takeaway

Mastering top, vmstat and iostat enables rapid identification of the true performance bottleneck on a Linux server. Follow the step‑by‑step workflow, verify the key metrics, locate the offending process, and apply the appropriate optimization – avoiding blind scaling or misdirected tuning.