Mastering Linux Disk I/O: Key Metrics and Essential Monitoring Tools

Linux's storage system I/O stack consists of three layers: the file system layer, the generic block layer, and the device layer. The generic block layer is the core of Linux disk I/O, providing a standard interface upward to file systems and applications and abstracting heterogeneous disks downward.

This article examines the key performance indicators of disks and how to view them.

Linux Disk Performance Metrics

When measuring disk performance, five common metrics are used: utilization, saturation, IOPS, throughput, and response time.

Utilization : the percentage of time the disk spends processing I/O. Values above 80% often indicate a bottleneck.

Saturation : the degree to which the disk is busy. At 100% saturation the disk cannot accept new I/O requests.

IOPS : the number of I/O requests per second.

Throughput : the amount of data transferred per second.

Response time : the interval between sending an I/O request and receiving its response.

Note: Utilization only considers whether I/O is occurring, not its size; a disk can still accept new I/O even when utilization reaches 100%.

When selecting a server for an application, you should benchmark the disk I/O performance under random read, sequential read, random write, and sequential write workloads with various I/O sizes (typically 512 B – 1 MB).

Disk I/O Observation

The most common tool for observing disk I/O performance is iostat, which reports metrics such as utilization, IOPS, and throughput derived from /proc/diskstats.

Example command:

# -d -x means display all disk I/O performance
$ iostat -d -x 1

Sample output:

Device            r/s   w/s   rkB/s   wkB/s  %util  r_await  w_await
loop0            0.00  0.00   0.00    0.00   0.00    0.00     0.00
sda               0.00  0.00   0.00    0.00   0.00    0.00     0.00

%util

shows disk I/O utilization. r/s and w/s represent read and write IOPS. rkB/s and wkB/s indicate throughput. r_await and w_await give response times.

Because iostat does not directly provide saturation, you can compare the observed average request queue length or wait time with benchmark results (e.g., from fio) to assess saturation.

Process I/O Observation

To view I/O per process, use pidstat or iotop.

Example pidstat command: $ pidstat -d 1 Sample output:

13:39:51      UID   PID  kB_rd/s  kB_wr/s  kB_ccwr/s  iodelay  Command
13:39:52      102  916    0.00     4.00      0.00        0    rsyslogd

pidstat

shows per‑process I/O details such as UID, PID, read/write rates, cancelled writes, and I/O delay measured in clock cycles.

Another useful tool is iotop, which sorts processes by I/O size similar to top:

$ iotop
Total DISK READ: 0.00 B/s | Total DISK WRITE: 7.85 K/s
  TID PRIO USER   DISK READ DISK WRITE SWAPIN  IO> COMMAND
15055 be/3 root   0.00 B/s   7.85 K/s   0.00 %  0.00 % systemd-journald

The first two lines show total disk read/write and actual disk read/write; differences arise from caching, buffering, and I/O merging. Subsequent columns provide thread ID, I/O priority, per‑second read/write sizes, swap‑in percentage, and I/O wait percentage.

Conclusion

The article introduced Linux disk I/O performance metrics—IOPS, throughput, utilization, saturation, and response time—and demonstrated how to retrieve them with iostat, as well as how to monitor per‑process I/O using pidstat and iotop. Effective analysis should consider read/write ratios, I/O types, and I/O sizes alongside these metrics.