Diagnosing Disk I/O Spikes That Freeze Linux Systems with iostat and iotop

1. Overview

The article starts with a typical SRE scenario: iowait spikes to 80%, simple commands like ls take seconds, and database slow‑query alerts flood the logs. The root cause is almost always disk I/O, but the problem is hidden across multiple layers of the Linux I/O stack.

2. Linux I/O Stack Full View

A request travels from the application (read/write/pread/pwrite/io_uring) through VFS + page cache, the filesystem (ext4/xfs/btrfs), the generic block layer (merge, scheduler, multi‑queue), the device driver (NVMe/SCSI/virtio‑blk) and finally the physical device (NVMe SSD, SATA SSD, HDD, cloud disk). Each layer can become a bottleneck, so troubleshooting must be performed layer by layer.

VFS + Page Cache: If free -h shows a tiny buff/cache or sar -B shows high pgpgin, reads are bypassing cache and hitting the disk.

Filesystem: Journaling (ext4) or log writes (xfs) add extra I/O; fragmentation can turn sequential reads into random reads. Use filefrag to inspect.

Block Layer: blk‑mq provides per‑CPU queues; merging, sorting and scheduling happen here.

Device Driver & Physical Device: NVMe has many hardware queues (64×64K depth) while SATA SSD has a single NCQ queue (depth 32), directly affecting concurrency.

3. I/O Scheduler Details

Four schedulers are available in kernel 6.x: none – best for NVMe (hardware handles scheduling). mq-deadline – general purpose, deadline‑based, good for SATA SSD and HDD. bfq – fair‑share for multi‑tenant HDD workloads. kyber – low‑latency SSD, auto‑adjusts queue depth.

Switch the scheduler at runtime with echo "bfq" > /sys/block/sda/queue/scheduler. Persist the choice with a udev rule.

4. iostat Deep Dive

iostat -xmt 1

shows per‑device metrics. Key fields: r/s, w/s – IOPS. rkB/s, wkB/s – throughput. rrqm/s, wrqm/s – merged requests (high values mean sequential or adjacent I/O).

r_await / w_await – average latency (most important; HDD normal 5‑15 ms, SSD 0.05‑0.5 ms). aqu-sz – average queue length. %util – useful for HDD, meaningless for SSD because of massive parallelism.

Deprecated svctm is unreliable on modern kernels and should be ignored.

Typical decision matrix (excerpt):

# await high + aqu-sz high + %util high → disk truly saturated (HDD)
# await high + aqu-sz low + %util low → single‑slow I/O (hardware or RAID)
# await normal + w/s very high + wkB/s low → many tiny writes, consider merging or raise dirty ratio
# rrqm/s ≈ 0 + rareq‑sz tiny → pure random reads, page cache ineffective
# w_await >> r_await → write bottleneck, check fsync or journal mode

5. iotop for Process‑Level I/O

After iostat confirms the disk is busy, sudo iotop -oP lists processes with active I/O. Example output shows MySQL reading 98 MiB/s and a backup tar reading 22 MiB/s, together saturating the bandwidth.

Key columns:

Total DISK READ/WRITE – all processes before page cache.

Actual DISK READ/WRITE – data that really hit the device.

IO> – percentage of time the process waited for I/O.

PRIO – I/O priority (best‑effort class 4 by default).

If a process cannot be killed, lower its priority with ionice -c 3 -p <PID>. Note that ionice works only with bfq or mq-deadline schedulers.

6. pidstat for Scriptable I/O Stats

Use pidstat -d 1 10 for interval‑based per‑process I/O, or pidstat -d -p <PID> 1 for a specific process. The kB_ccwr/s column shows cancelled writes (written to cache then overwritten).

7. blktrace + btt for Block‑Layer Analysis

Run sudo blktrace -d /dev/sda -w 10 -o trace to capture the full lifecycle (Q, G, I, D, C). Parse with blkparse -i trace -d trace.txt. Example line shows a read request of 8 sectors (4 KB) from mysqld with total latency 850 µs (Q→C).

Use btt -i trace.bin to get aggregated latency distribution:

# Q2C  min=0.000085  avg=0.001250  max=0.025000  N=12847
# Q2D  min=0.000005  avg=0.000018  max=0.000350  N=12847
# D2C  min=0.000080  avg=0.001232  max=0.024800  N=12847

If Q2D dominates, the scheduler is the bottleneck; if D2C dominates, the hardware is the limit.

8. Visualising I/O with iowatcher

Install iowatcher and generate an SVG: iowatcher -t trace -o io-pattern.svg. The chart shows IOPS, throughput and latency distribution over time.

9. fio for Disk Benchmarking

Typical benchmark commands (using io_uring engine):

# Random read 4 KB, depth 64, 4 jobs
fio --name=rand-read --ioengine=io_uring --rw=randread \
    --bs=4k --iodepth=64 --numjobs=4 --size=4G \
    --direct=1 --runtime=60 --group_reporting \
    --filename=/dev/nvme0n1

# Sequential read 1 MiB, depth 16
fio --name=seq-read --ioengine=io_uring --rw=read \
    --bs=1m --iodepth=16 --numjobs=1 --size=8G \
    --direct=1 --runtime=60 --group_reporting \
    --directory=/mnt/test

Sample output shows IOPS=185.2k, await≈0.08 ms for a high‑end NVMe, confirming the device’s capability.

Comparing libaio vs io_uring shows a 10‑30 % performance gain for io_uring in high‑IOPS workloads.

10. Filesystem Choice and Mount Options

Table summary (converted to text):

ext4: max file size 16 TB, ordered journal (default), no snapshot.
xfs: max file size 8 EB, excellent concurrent write performance, no online shrink.
btrfs: max file size 16 EB, native snapshots & compression, slower small‑file performance.

Recommendation:
- Database servers (MySQL/PostgreSQL): xfs (good concurrent writes, default on RHEL).
- General Linux servers: ext4 (stable, most documentation).
- Snapshot/compression needed (logs, containers): btrfs.
- Kubernetes nodes: xfs (overlayfs works best).

Mount options for performance:

# ext4 high‑performance mount
mount -o noatime,nodiratime,barrier=0,data=writeback /dev/sda1 /data

# xfs high‑performance mount
mount -o noatime,logbufs=8,logbsize=256k /dev/sda1 /data

11. I/O Tuning

Readahead : Adjust with blockdev --setra. Use 1 MiB for sequential workloads (Kafka, HDFS) and 32 KB for random‑read workloads (OLTP).

Dirty Ratio (PageCache write‑back):

# Database (low latency)
sysctl -w vm.dirty_ratio=5
sysctl -w vm.dirty_background_ratio=2
sysctl -w vm.dirty_expire_centisecs=1000
sysctl -w vm.dirty_writeback_centisecs=100

# Log/streaming (throughput)
sysctl -w vm.dirty_ratio=40
sysctl -w vm.dirty_background_ratio=20
sysctl -w vm.dirty_expire_centisecs=6000
sysctl -w vm.dirty_writeback_centisecs=500

Persist in /etc/sysctl.d/60-io-tuning.conf.

Queue Depth & Merge : Increase /sys/block/sda/queue/nr_requests to 1024 for high concurrency; set /sys/block/sda/queue/nomerges=2 for pure random I/O.

cgroup v2 I/O Limits (multi‑tenant containers):

# Create cgroup
mkdir -p /sys/fs/cgroup/backup-jobs
echo "+io" > /sys/fs/cgroup/backup-jobs/cgroup.subtree_control
# Limit to 50 MiB/s and 1000 IOPS on /dev/sda (8:0)
echo "8:0 rbps=52428800 wbps=52428800 riops=1000 wiops=1000" > /sys/fs/cgroup/backup-jobs/io.max
# Add process
echo $BACKUP_PID > /sys/fs/cgroup/backup-jobs/cgroup.procs

12. Common Troubleshooting Cases

Case 1 – Nightly backup stalls database : iostat shows w_await jump, iotop reveals a rsync job reading 160 MiB/s. Solution – lower backup priority with ionice -c 3 nice -n 19 rsync … or cgroup bandwidth limit.

Case 2 – ext4 journal write amplification : High w/s but wareq‑sz only 4 KB. blktrace shows many writes from jbd2. Switch mount option to data=ordered (or writeback if UPS is present).

Case 3 – NVMe %util always 100 % : iostat shows await=0.08 ms and IOPS far below device rating. Explanation – %util is misleading for multi‑queue devices; monitor await and actual IOPS instead.

13. bpftrace for Latency Distribution

Use the built‑in biolatency-bpfcc -D 1 to get per‑disk latency histograms. For per‑process latency, write a custom script (saved as io-latency-by-process.bt) that records block_rq_issue timestamps and computes histograms on block_rq_complete. The output groups latency by process name, exposing long‑tail delays.

14. Summary

The article emphasizes a disciplined, layered approach: start with top → iostat → iotop → blktrace/btt → bpftrace → tuning (scheduler, readahead, dirty ratio, cgroup). Understanding the meaning of each metric (especially await vs %util) prevents mis‑diagnosis. Proper scheduler selection, kernel‑level parameters, and filesystem mount options together can eliminate most I/O‑related stalls without hardware upgrades.

15. Further Learning

eBPF/bpftrace deep‑dive (Brendan Gregg’s BPF Performance Tools).

io_uring programming and performance characteristics.

NVMe advanced features (multi‑queue mapping, namespaces, ZNS).

Building end‑to‑end storage observability with Prometheus, node_exporter diskstats, and Grafana dashboards.