Comprehensive Guide to Storage I/O Performance: Concepts, Monitoring, and Optimization Across System Layers

Whether you are a system operations engineer or a solution support specialist, performance is the most critical aspect of your work; this series aims to start from basic concepts and comprehensively summarize disk, storage, and operating system layers to give a thorough understanding of I/O performance concepts, monitoring, and tuning.

The article begins by describing the flow of I/O (Block) and its relationship to storage performance, tracing the journey from the I/O origin through various nodes that affect performance, and concludes with a SQL Server application example.

Different applications exhibit distinct storage I/O access patterns; the article outlines typical I/O types for various workloads, providing data useful for storage simulation and stress‑testing.

Subsequent sections analyze I/O aggregation, write penalties, common business models, bandwidth and utilization calculations, read/write ratios, RAID penalties, and the impact of random versus sequential I/O and caching on storage performance.

The detailed analysis of disk I/O includes addressing, rotation, and transfer effects, offering calculation methods from the disk’s perspective.

Each layer—host application, operating system, file system, page cache, generic block, I/O scheduler, and block device driver—is examined for its characteristics and optimization opportunities.

Host Application Layer : Discusses how sequential and contiguous I/O from the host improves merging efficiency in storage arrays, with examples such as OLTP, OLAP, VDI, and SPC‑1 workloads.

Operating System Layer : Explains how OS scheduling and optimizations can make actual IOPS exceed theoretical limits, and outlines the Linux I/O path from VFS through file system, cache, generic block, scheduler, driver, to the physical device.

Virtual File System (VFS) Layer : Describes VFS as an abstraction that unifies different file systems, detailing its core structures—superblock, inode, directory entry, and file object.

EXT2 File System : Covers block sizes, inode tables, group descriptors, and how data is stored across blocks, including direct, indirect, double‑indirect, and triple‑indirect pointers.

Page Cache Layer : Highlights the role of page and buffer caches in reducing disk access latency through prefetching (synchronous and asynchronous) and write‑back mechanisms.

Generic Block Layer : Details how this layer abstracts hardware characteristics, manages block‑size alignment, and utilizes DMA (including scatter/gather) for efficient data transfer.

I/O Scheduler Layer : Discusses request queue management, merging, and common scheduling algorithms such as Noop, CFQ, Deadline, and AS.

Block Device Driver Layer : Briefly notes that drivers translate I/O requests into commands for physical block devices.

The article then presents design techniques based on disk I/O characteristics, such as using append‑only writes to achieve high sequential write performance, and the trade‑offs for read latency.

Examples include HDFS’s write‑once‑read‑many model, Kafka’s log‑structured append‑only design with segment indexing, and the use of LSM‑trees in NoSQL databases to convert random writes into sequential writes, improving write throughput at the cost of some read performance.

Further, it discusses file merging and metadata optimization for handling massive small files, emphasizing strategies like consolidating small files into larger ones, simplifying metadata, and employing side‑databases or embedded metadata to reduce open‑call overhead and improve locality.

Overall, the guide provides practical insights for architects and engineers to design storage‑aware systems that maximize I/O efficiency across the full software stack.