How Traditional System Call I/O Works and How to Optimize It

Traditional System Call I/O

Read operation

Write operation

Network I/O

Disk I/O

High‑performance I/O optimizations

Storage device I/O stack

I/O Buffering

Traditional System Call I/O

In Linux, the classic way to access data is through the read() and write() system calls: read() copies data from a file into a user‑space buffer, and write() sends data from the buffer to a network port.

The diagram below shows that a traditional I/O operation involves two CPU copies, two DMA copies (four copies total), and four context switches.

CPU copy : the CPU directly moves data, consuming CPU cycles.

DMA copy : the CPU instructs the DMA controller to transfer data, freeing the CPU.

Context switch : the CPU switches from user mode to kernel mode on entry, and back on return.

Read operation

If the requested data is already in the process's page cache, it is read directly from memory. Otherwise the kernel loads the data from disk into a read buffer, then copies it to the user buffer.

read(file_fd, tmp_buf, len);

Using the traditional I/O path, a read() triggers two context switches, one DMA copy and one CPU copy.

The read flow is:

The user process calls read(), causing a switch from user space to kernel space.

The CPU, via DMA, copies data from memory or disk into the kernel’s read buffer.

The CPU copies data from the read buffer to the user buffer.

The context switches back to user space and read() returns.

Write operation

When an application calls write() to send data, the data is first copied from the user buffer to the kernel’s socket buffer, then the NIC transmits it.

The traditional write path triggers two context switches, one CPU copy and one DMA copy.

The write flow is:

The user process calls write(), switching to kernel space.

The CPU copies data from the user buffer to the kernel’s socket buffer.

The CPU uses DMA to move data from the socket buffer to the NIC.

The context switches back to user space and write() returns.

Network I/O

Disk I/O

High‑performance I/O Optimizations

Zero‑copy techniques.

Multiplexing (I/O multiplexing) techniques.

PageCache optimizations.

The PageCache is the OS’s file cache that reduces disk I/O by keeping file data in memory pages, allowing reads and writes to approach memory speed.

PageCache read strategy : when a process issues a read(), the kernel first checks the PageCache.

If the data is present, it is returned directly without disk access.

If not, the kernel schedules a block I/O to fetch the data, reads a few pages (typically three) into the PageCache, and then serves the request.

PageCache write strategy : a write() first writes to the PageCache and marks the page as “dirty”. A background flusher later writes dirty pages to disk, clearing the dirty flag. Dirty pages are flushed when:

Free memory falls below a threshold.

A page has been dirty for longer than a configured interval.

The process calls sync() or fsync().

Storage Device I/O Stack

The Linux I/O stack consists of three layers below the system‑call interface:

File system layer : for a write(), the kernel copies user data into the file‑system cache and eventually syncs it down.

Block layer : manages block‑device I/O queues, merges and schedules requests (recall OS I/O scheduling algorithms).

Device layer : interacts with the hardware via DMA to transfer data.

Understanding this stack helps relate Buffered I/O, mmap, and Direct I/O to their positions in the Linux I/O hierarchy.

Linux I/O system : Traditional Buffered I/O reads a cold file by opening it, checking the PageCache, creating a cache entry if missing, queuing a block I/O request, and finally using DMA to fetch the sector into the cache before copying to the user buffer.

The process involves multiple copies: from disk to PageCache (first copy) and from PageCache to user buffer (second copy). mmap maps the PageCache directly into user space, eliminating the second copy.

Direct I/O bypasses the PageCache, moving data directly between user space and the block device via DMA, which reduces copies for writes and can speed up reads for the first access, though subsequent reads may suffer without a cache.

Both mmap and Direct I/O require page‑aligned buffers and I/O sizes that are multiples of the device’s block size (older kernels even required multiples of the filesystem block size).

I/O Buffering

When a program calls file‑operation functions, user data flows through several buffering layers before reaching the disk, as illustrated in the diagram. The C standard I/O library provides its own user‑space buffers to reduce the number of system calls. The kernel maintains a buffer cache (PageCache) to further reduce disk accesses. PageCache stores file contents, while the lower‑level Buffer Cache stores raw device blocks.