Understanding Linux I/O: Storage Hierarchy, Page Cache, and Performance Considerations

Preface

Before the formal discussion, several questions are raised: the difference between HDD and SSD, whether multithreaded writes can accelerate file writing, if a successful write guarantees data is on disk, the atomicity of write , the need for file locking, and the performance claims of mmap . The article is intended for readers who have already studied Linux fundamentals and have some engineering experience.

Memory Hierarchy Pyramid

The storage hierarchy of modern computers forms a pyramid: higher levels (registers, CPU caches) are faster but more expensive and smaller, while lower levels (DRAM, local disks) are slower, cheaper, and larger. Locality of reference allows this cost‑effective design to achieve good performance.

This article focuses on local disk I/O and its interaction with DRAM.

Ubiquitous Caching

When a program invokes file‑operation functions, user data passes through several caching layers before reaching the disk. The diagram (omitted) shows the relationship between user‑space stdio buffers, the kernel’s page cache, and the device’s buffer cache.

stdio provides user‑space buffering to reduce expensive system calls; functions like fflush , setbuf control this buffer. The kernel maintains a Kernel buffer cache , commonly called the Page Cache , while the device’s own cache is the Buffer Cache . The page cache stores file contents, whereas the buffer cache stores raw block data independent of any filesystem.

Linux Kernel I/O Stack

The Linux I/O stack consists of three main layers:

File‑system layer – e.g., write copies user data into the file‑system cache.

Block layer – manages I/O queues, merges and orders requests (the classic OS I/O scheduling algorithms).

Device layer – uses DMA to transfer data between memory and the physical device.

These layers explain where Buffered I/O , mmap , and Direct I/O fit in the stack.

Page Cache Synchronization

Cache synchronization can be either Write‑Through (data is written to the page cache and immediately flushed to the lower layer) or Write‑Back (data is considered written once it reaches the page cache; the actual disk write happens asynchronously). Linux’s default for buffered I/O is write‑back.

Dirty pages are flushed to disk by the pdflush kernel thread based on memory pressure, time‑out thresholds, or explicit calls to sync , fsync , and fdatasync . The flushing behavior can be forced to write‑through by opening a file with the O_SYNC flag.

File Operations and Locks

The write system call is not atomic; concurrent writes to the same region can interleave, making the code unreliable without additional synchronization. The only kernel‑guaranteed atomic operations are open with O_CREAT and O_APPEND . While O_APPEND moves the file offset atomically, the atomicity of the actual write is still debated.

Linux provides two file‑locking mechanisms: flock (BSD style) and fcntl (System V style). In practice, many designs avoid shared‑file writes by using process‑level mutexes or by relying on databases that implement their own locking.

Disk Performance Testing

Understanding a machine’s I/O capabilities (IOPS, latency, throughput) is essential for system design. Mechanical HDDs typically top out at ~120 MiB/s sequentially, while modern SSDs can exceed 2 GiB/s. Random access is limited by seek time on HDDs but is comparable to sequential on SSDs.

Multi‑threaded I/O can improve SSD throughput because SSDs handle many concurrent requests (high I/O depth). Tools like fio are commonly used to benchmark storage devices.

Conclusion

The article offers a concise yet thorough introduction to Linux I/O mechanisms, encouraging readers to explore system‑call internals, caching behavior, and performance implications in depth.

References

[1] Computer Systems: A Programmer's Perspective, Chapter 6. [2] Locality of reference – Wikipedia. [3] The Linux Programming Interface, Chapter 13. [4] Linux Storage Stack Diagram – Thomas‑Krenn.com. [5] Detailed explanation of O_DIRECT and O_SYNC. [6] Discussion on O_APPEND atomicity. [7] Coding for SSD – dirtysalt.github.io. [8] fio – Linux I/O benchmarking tool. [9] How to benchmark disk I/O – BinaryLane. [10] 深入Linux内核架构 – 莫尔勒.