How to Master Linux Disk I/O Performance: Metrics, Tools, and Best Practices

Linux's storage I/O stack consists of three layers: the file system layer, the generic block layer, and the device layer.

The generic block layer is the core of Linux disk I/O. It provides a standard interface to block devices for file systems and applications, and abstracts heterogeneous disk devices into a unified block device.

Linux Disk Performance Metrics

When measuring disk performance, five common metrics are used: utilization, saturation, IOPS, throughput, and response time.

Utilization : the percentage of time the disk spends processing I/O. Values above 80% often indicate a performance bottleneck.

Saturation : the degree to which the disk is busy handling I/O. At 100% saturation the disk cannot accept new I/O requests.

IOPS : the number of I/O operations per second.

Throughput : the amount of data transferred per second.

Response time : the interval between sending an I/O request and receiving its response.

Note: Utilization only reflects whether I/O is occurring, not the size of the I/O; a disk can still accept new requests even at 100% utilization.

When selecting a server for an application, benchmark the disk I/O performance under various workloads (random read, sequential read, random write, sequential write) and different I/O sizes (typically 512 B – 1 MiB).

Disk I/O Observation

The most common tool for observing disk I/O is iostat, which reports metrics such as utilization, IOPS, and throughput derived from /proc/diskstats.

Example output of iostat -d -x 1:

# -d -x means display all disk I/O performance
$ iostat -d -x 1
Device            r/s   w/s   rkB/s   wkB/s  %rrqm %wrqm r_await w_await aqu-sz raq-sz waq-sz svctm %util
loop0            0.00  0.00    0.00    0.00   0.00  0.00   0.00    0.00   0.00   0.00   0.00   0.00  0.00   0.00
loop1            0.00  0.00    0.00    0.00   0.00  0.00   0.00    0.00   0.00   0.00   0.00   0.00  0.00   0.00
sda              0.00  0.00    0.00    0.00   0.00  0.00   0.00    0.00   0.00   0.00   0.00   0.00  0.00   0.00
sdb              0.00  0.00    0.00    0.00   0.00  0.00   0.00    0.00   0.00   0.00   0.00   0.00  0.00   0.00

Key fields to note: %util – disk I/O utilization. r/s and w/s – IOPS for reads and writes. rkB/s and wkB/s – throughput for reads and writes. r_await and w_await – average read and write response times.

Measuring saturation directly is difficult; instead, compare observed average queue length or wait time with benchmark results (e.g., from fio) to assess saturation.

Process I/O Observation

To view per‑process I/O, use pidstat or iotop. Example pidstat -d 1 output:

$ pidstat -d 1
13:39:51      UID   PID  kB_rd/s  kB_wr/s  kB_ccwr/s  iodelay  Command
13:39:52      102   916    0.00     4.00      0.00       0   rsyslogd

Fields provided by pidstat include:

User ID (UID) and Process ID (PID).

Data read per second ( kB_rd/s).

Data written per second ( kB_wr/s).

Cancelled write data per second ( kB_ccwr/s).

Block I/O delay ( iodelay) measured in clock cycles. iotop sorts processes by I/O size, similar to top. Example output:

$ iotop
Total DISK READ: 0.00 B/s | Total DISK WRITE: 7.85 K/s
Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 0.00 B/s
  TID PRIO USER   DISK READ DISK WRITE SWAPIN  IO> COMMAND
15055 be/3 root   0.00 B/s   7.85 K/s   0.00 %  0.00 % systemd-journald

The first two lines show total requested and actual disk read/write sizes; differences arise from caching, buffering, and I/O merging. Subsequent columns display thread ID, I/O priority, per‑second read/write sizes, swap‑in percentage, and I/O wait percentage.

Conclusion

This article introduced Linux disk I/O performance metrics—IOPS, throughput, utilization, saturation, and response time—and demonstrated how to retrieve them with iostat, as well as how to monitor per‑process I/O using pidstat and iotop. Effective analysis should also consider read/write ratios, I/O types, and I/O sizes.