Master Linux Disk I/O: Key Metrics, Tools, and Real-World Troubleshooting

1 Disk Performance Metrics

Disk performance metrics include five common indicators: utilization, saturation, IOPS, throughput, and response time. These are the basic standards for measuring disk performance and are used in I/O troubleshooting.

Utilization: percentage of time the disk handles I/O, ignoring throughput. Even at 100% utilization, the disk may accept new I/O if no latency.

Saturation: degree of busy I/O handling; at 100% saturation the disk cannot accept new I/O.

IOPS (Input/Output Per Second): number of I/O requests per second; mechanical drives ~100 IOPS.

Throughput: amount of data transferred per second (e.g., SATA3.0 HDD ~200 MB/s, SSD ~400 MB/s, PCIe SSD ~1.2 GB/s). Reducing seek can increase throughput.

Response time: interval from request issuance to response; caches in MySQL, file systems affect it.

These metrics are frequently discussed when evaluating disk performance, but analysis must consider read/write ratios, I/O type (random vs sequential), I/O size, and parallelism.

Random workloads rely on IOPS; sequential workloads rely on throughput. Examples: application startup reads 80% of data; databases with many random reads benefit from IOPS; multimedia with sequential reads benefits from throughput; SSDs show similar performance for random and sequential I/O, though small random writes can cause write amplification.

2 Benchmark Testing

Before selecting servers, benchmark disk I/O using tools like fio to measure IOPS, throughput, and response time, adapting to the application’s I/O characteristics (e.g., various block sizes from 512 B to 1 MB, random/seq read/write).

Typical benchmark results include maximum throughput, maximum IOPS, max random read IOPS, sequential read response time, and random read response time.

3 Disk I/O Observation

The iostat tool reports per‑disk utilization, IOPS, throughput, and other metrics derived from /proc/diskstats.

[sdk_test@ssdk1 server]$ iostat -d -x -k 1 2
Linux 2.6.32-431.11.15.el6.x86_64 (ssdk1) 10/14/2016 _x86_64    (4 CPU)

Device:    rrqm/s wrqm/s   r/s   w/s  rkB/s  wkB/s avgrq-sz avgqu-sz await svctm %util
vda          0.00   0.19  0.00  0.65   0.04   3.37   10.41     0.00  0.78 0.41 0.03
vdb          0.00   5.85  0.29  1.13   6.23  27.93   48.06     0.00  1.44 0.41 0.06

Device:    rrqm/s wrqm/s   r/s   w/s  rkB/s  wkB/s avgrq-sz avgqu-sz await svctm %util
vda          0.00   0.00  0.00  0.00   0.00   0.00    0.00     0.00  0.00 0.00 0.00
vdb          0.00   0.00  0.00  0.00   0.00   0.00    0.00     0.00  0.00 0.00 0.00

Key fields:

%util – disk I/O utilization.

r/s + w/s – IOPS.

rkB/s + wkB/s – throughput.

r_await + w_await – response time.

avgrq‑sz – average request size (sectors).

avgqu‑sz – average queue length (shorter is better).

Combine these with average request sizes (rareq‑sz, wareq‑sz) for a full picture; iostat does not directly show saturation.

Since Linux 4.20, /proc/pressure/io provides I/O pressure statistics:

cat /proc/pressure/io
some  avg10=0.00 avg60=0.00 avg300=0.00 total=56385169
full  avg10=0.00 avg60=0.00 avg300=0.00 total=54915860

‘some’ indicates the percentage of time at least one task waited for I/O; ‘full’ indicates all tasks waited. The avg fields represent the last 10, 60, and 300 seconds.

4 Process I/O Observation

To see which processes generate I/O, use pidstat -d or iotop.

$ pidstat -d 1
13:39:51      UID   PID kB_rd/s kB_wr/s kB_ccwr/s iodelay Command
13:39:52      102   996   0.00   10.00   0.00      0   rsyslogd

pidstat reports UID, PID, read/write rates, cancelled writes, I/O delay, and command name.

$ iotop
Total DISK READ:       0.00 B/s | Total DISK WRITE:       7.85 K/s
Actual DISK READ:      0.00 B/s | Actual DISK WRITE:      0.00 B/
  TID PRIO USER      DISK READ DISK WRITE SWAPIN  IO> COMMAND
15055 be/3 root      0.00 B/s   7.85 K/s   0.00 %   0.00 % systemd-journald

iotop shows per‑process disk read/write rates, swap‑in percentage, and I/O wait percentage.

5 Case Study: I/O Spike Investigation

When an I/O spike alert occurs, start with top to check CPU wait (wa) time, which often indicates disk I/O issues.

Next, run iostat -x to identify the busy disk.

Use iotop to pinpoint the offending process.

The investigation revealed that the jbd2 journaling process caused the high write I/O. Disabling jbd2 in a stateless Kubernetes service resolved the issue (not applicable to all scenarios).

6 Summary

We reviewed Linux disk I/O metrics (IOPS, throughput, utilization, saturation, response time) and tools (iostat, PSI, pidstat, iotop). Effective analysis combines these metrics with read/write ratios, I/O type, and request size. A real‑world case demonstrated locating an I/O spike to a specific process and mitigating it.