How Zero‑Copy, PageCache, and Async I/O Supercharge File Transfer Performance

Traditional Buffering Approach

Servers typically read a file from disk, allocate a small user‑space buffer (e.g., 32 KB), split a large file (e.g., 320 MB) into many chunks, and repeatedly call read and write to send each chunk to the client.

This method incurs two major performance penalties.

Why It Performs Poorly

Excessive Context Switches

Each 32 KB chunk requires a read system call and a write system call, causing a user‑to‑kernel and kernel‑to‑user transition. Processing 32 KB therefore triggers four context switches; repeating this 10 000 times results in about 40 000 switches, which, although each is only microseconds, adds up significantly under high concurrency.

Redundant Memory Copies

The same 32 KB buffer is copied from kernel to user space and back, leading to roughly four times the original data being copied (320 MB becomes 1.28 GB). This unnecessary copying wastes CPU cycles and reduces the server’s ability to handle concurrent requests.

Zero‑Copy: Reducing Switches and Copies

Zero‑copy combines the two system calls into one kernel‑space operation that moves data directly from the file’s page cache to the socket buffer, eliminating the user‑space buffer.

Benefits include:

Only two context switches per transfer (one for the combined operation, one for completion notification).

Three memory copies instead of four, cutting total copied data roughly in half.

If the network card supports Scatter‑Gather DMA (SG‑DMA), the socket buffer copy can also be removed, leaving just two memory copies.

PageCache: The OS Disk Cache

When a file is read, the kernel first copies the data into the PageCache, then serves it to the requesting process. PageCache improves performance by:

Replacing slow disk reads with fast memory reads (leveraging temporal locality).

Prefetching subsequent blocks to hide disk‑seek latency.

However, for very large files, PageCache can become a liability: it occupies cache space, evicts hot small files, and adds an extra copy step that may outweigh its benefits.

When to Bypass PageCache

In high‑concurrency scenarios with large files, it is better to avoid PageCache and use zero‑copy or direct I/O, because the large file’s data is unlikely to be re‑accessed soon, and caching it would waste memory and CPU.

Asynchronous I/O + Direct I/O

Asynchronous I/O (AIO) decouples the read request from waiting for data, allowing the process to continue other work while the kernel fetches data. AIO alone does not use PageCache; it works with Direct I/O, which bypasses the cache entirely.

Direct I/O is useful when:

The application already implements its own caching (e.g., databases like MySQL).

Large files are transferred under high load, where PageCache would cause extra copies and memory pressure.

The downside is that Direct I/O forfeits kernel optimizations such as request merging and read‑ahead prefetching.

Practical Strategy

Combine the techniques based on file size:

Use zero‑copy for small‑to‑medium files to minimize context switches and copies.

Employ asynchronous Direct I/O for large files to avoid blocking and cache pollution.

Configuration parameters (e.g., Nginx’s directio directive) can define the size threshold.

Conclusion

By reducing system‑call frequency, cutting memory copies, and intelligently choosing between zero‑copy, PageCache, asynchronous I/O, and Direct I/O, file‑transfer throughput can more than double, latency drops, and CPU usage is lowered, enabling higher request concurrency.