Mastering iostat: How to Diagnose Linux I/O Bottlenecks

Linux systems can experience performance issues; commands like top, iostat, free, and vmstat help locate problems. iostat provides detailed I/O statistics.

1. Basic Usage of iostat

1. Command Introduction

Example: iostat -d -k 1 10 -d shows device (disk) usage.

-k forces kilobytes unit.

1 10 means refresh every 1 second, total 10 times.

2. Usage Demonstration

# iostat -x 1 10
Linux 2.6.18-92.el5xen 02/03/2009
avg-cpu: %user %nice %system %iowait %steal %idle
 1.10 0.00 4.82 39.54 0.07 54.46
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 3.50 0.40 2.50 5.60 48.00 18.48 0.00 0.97 0.97 0.28
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sde 0.00 0.10 0.30 0.20 2.40 2.40 9.60 0.00 1.60 1.60 0.08
sdf 17.40 0.50 102.00 0.20 12095.20 5.60 118.40 0.70 6.81 2.09 21.36
sdg 232.40 1.90 379.70 0.50 76451.20 19.20 201.13 4.94 13.78 2.45 93.16

3. Parameter Explanation

Key fields:

rrqm/s – reads merged per second.

wrqm/s – writes merged per second.

r/s – reads per second.

w/s – writes per second.

rsec/s – sectors read per second.

wsec/s – sectors written per second.

rkB/s – kilobytes read per second (half of rsec/s).

wkB/s – kilobytes written per second (half of wsec/s).

avgrq-sz – average request size (sectors).

avgqu-sz – average queue length.

await – average wait time (ms).

svctm – average service time (ms).

%util – percentage of time the device was busy.

Interpretation tips:

If %util approaches 100%, the device is saturated.

await > 70 ms indicates high I/O pressure.

Combine with vmstat’s bi and wa to assess waiting processes and CPU I/O wait.

svctm < await is typical; svctm reflects disk performance, await reflects queue length and request pattern.

When svctm ≈ await, I/O has little wait; when await » svctm, the queue is long and response slows.

Consider faster disks, kernel elevator tuning, application optimization, or CPU upgrade if response exceeds acceptable limits.

2. Understanding I/O with Real‑World Analogy

1. Supermarket Checkout Analogy

Comparing I/O metrics to a checkout line helps visualize r/s+w/s as total customers, avgqu‑sz as average line length, svctm as cashier speed, await as average waiting time, avgrq‑sz as average basket size, and %util as proportion of time the line is occupied.

2. Parameter Case Study

# iostat -x 1
avg-cpu: %user %nice %sys %idle
16.24 0.00 4.31 79.44
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq‑sz avgqu‑sz await svctm %util
/dev/vda 0.00 44.90 1.02 27.55 8.16 579.59 4.08 289.80 20.57 22.35 78.21 5.00 14.29
/dev/vdb 0.00 44.90 1.02 27.55 8.16 579.59 4.08 289.80 20.57 22.35 78.21 5.00 14.29
/dev/vbc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

Analysis:

Total I/O operations ≈ 28.57 /s (mostly writes).

Average service time ≈ 5 ms, but average wait ≈ 78 ms because many requests arrive simultaneously.

Queue length is low (~2), indicating bursty arrivals with idle periods.

I/O busy time is 14.29 % (≈85 % idle).

Reported avgqu‑sz of 22.35 appears erroneous; correct value should be ~2.23.

3. Advanced iostat Usage

1. Overview

iostat reports CPU, device, and terminal statistics. By default it shows tdc (terminal, disk, CPU). Options replace the default set.

2. Syntax

iostat <options> interval count

Options select devices (‑d, ‑c, ‑t, ‑tdc) and ‑x for extended statistics.

3. Example

$ iostat -xtc 5 2
extended disk statistics tty cpu
disk r/s w/s Kr/s Kw/s wait actv svc_t %w %b tin tout us sy wt id
sd0 2.6 3.0 20.7 22.7 0.1 0.2 59.2 6 19 0 84 3 85 11 0
sd1 4.2 1.0 33.5 8.0 0.0 0.2 47.2 2 23
sd2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
sd3 10.2 1.6 51.4 12.8 0.1 0.3 31.2 3 31

4. Parameter Details

disk – device name.

r/s – reads per second.

w/s – writes per second.

Kr/s – kilobytes read per second.

Kw/s – kilobytes written per second.

wait – average queue length.

actv – average active requests.

%w – percent of time requests waited (queue non‑empty).

%b – percent of time device was busy.

5. Recommendations

Watch for high reads/writes, %b > 5 %, and svc_t > 30 ms.

Optimize applications, enable caching, stripe across multiple disks.

Increase system parameters such as inode cache.

Migrate file systems to faster disks or controllers.