IO Performance Evaluation, Tools, Metrics, and Optimization Strategies

In production environments, long I/O latency caused by issues such as switch failures, aging cables, insufficient storage stripe width, cache shortage, QoS limits, or improper RAID settings can lead to reduced throughput and slow response times.

1. Prerequisite for evaluating I/O capability

Understanding the system's I/O model is essential. An I/O model describes the mix of read/write ratios, I/O sizes, and access patterns for a specific workload, which is the basis for capacity planning and problem analysis.

(1) I/O model

The basic model includes IOPS, bandwidth, and I/O size. For disk I/O, additional factors to consider are:

Which disks handle the I/O?

Read‑to‑write ratio.

Whether reads are sequential or random.

Whether writes are sequential or random.

(2) Why refine the I/O model?

Different models yield different maximum values for IOPS, bandwidth (MB/s), and response time. For example, testing with random small I/O shows lower bandwidth but higher latency, while sequential large I/O shows higher bandwidth but lower IOPS. Therefore, capacity planning and performance tuning must be based on the actual I/O model of the workload.

2. Evaluation tools

(1) Disk I/O tools

Common tools include orion, iometer, dd, xdd, iorate, iozone, postmark . They differ in OS support and simulation capabilities. For instance, Orion simulates Oracle database I/O using the same software stack as Oracle.

(2) Network I/O tools

Ping : basic latency test with configurable packet size.

IPerf, ttcp : measure maximum TCP/UDP bandwidth, latency, and packet loss.

Windows‑specific tools: NTttcp, LANBench, pcattcp, LAN Speed Test (Lite), NETIO, NetStress .

3. Main monitoring metrics and common tools

(1) Disk I/O

For Unix/Linux, Nmon and iostat are widely used. Nmon is good for post‑analysis, while iostat provides real‑time data.

IOPS: Total IOPS – Nmon DISK_SUMM Sheet: IO/Sec Read IOPS per disk – Nmon DISKRIO Sheet Write IOPS per disk – Nmon DISKWIO Sheet Command line: iostat -Dl – tps, rps, wps

Bandwidth: Total – Nmon DISK_SUMM Sheet: Disk Read KB/s, Disk Write KB/s Read bandwidth per disk – Nmon DISKREAD Sheet Write bandwidth per disk – Nmon DISKWRITE Sheet Command line: iostat -Dl – bps, bread, bwrtn

Response time: Read latency – iostat -Dl – read‑avg‑serv, max‑serv Write latency – iostat -Dl – write‑avg‑serv, max‑serv

Other: disk busy degree, queue depth, queue full count, etc.

(2) Network I/O

Bandwidth – best measured on the network device (e.g., Nmon NET Sheet or topas Network: BPS, B‑In, B‑Out ).

Latency – simple ping can show round‑trip time, but for precise measurement capture TCP SYN/SYN‑ACK timing or use dedicated network probes.

4. Performance positioning and optimization

(1) Host side – If host latency is high while storage latency is low, investigate application‑level I/O, OS parameters (queue depth, driver limits), and hardware (HBA, DMA size).

(2) Network side – When host latency is high and storage latency is low, examine bandwidth saturation, switch configuration, faulty cables, or multi‑path routing issues.

(3) Storage side – If both host and storage latencies are high, analyze storage configuration: RAID level, stripe width/depth, cache size, LUN type (thin vs thick), QoS limits, and ongoing background tasks such as snapshots or rebuilds.

5. Low‑latency transaction / high‑speed trading I/O tuning

Business perspective – reduce or eliminate unnecessary logging; adjust log levels.

Storage media – use SSDs, SSD cache, tiered storage, RAMDISK, or increase cache on storage servers.

Configuration – choose appropriate RAID (e.g., RAID10), ensure sufficient stripe depth and width.

I/O path – adopt high‑speed networking (avoid low‑speed iSCSI).

6. Network I/O problem locating methods

Use packet capture tools to identify where latency or packet loss occurs, and optionally run IPtrace on the host for deeper analysis.

7. Cases misidentified as I/O problems

Examples include Oracle buffer‑busy waits caused by index contention and intermittent ping delays caused by CPU oversubscription on heavily partitioned LPARs, illustrating that apparent I/O slowness can stem from database design or OS scheduling issues.

For more detailed performance tuning techniques, refer to the “IO Knowledge and System Performance Deep Tuning (2nd Edition)” ebook linked in the original article.