Boost File Transfer Speed: Zero‑Copy, PageCache, Async & Direct I/O Explained

How is file transfer typically implemented?

When a server provides file transfer, it reads the file from disk and sends it over the network. A naïve implementation reads the file in small chunks (e.g., 32 KB) into a user‑space buffer and writes each chunk to the socket, repeating this thousands of times for large files.

This approach suffers from two main performance issues.

Context switches: Each 32 KB chunk requires a read system call and a write system call, causing a user‑to‑kernel and kernel‑to‑user transition. Processing 10 000 chunks results in about 40 000 context switches, which, while individually cheap, add up significantly under high concurrency.

Memory copies: The same 40 000 copies double the data transferred (320 MB file becomes 1.28 GB copied), wasting CPU cycles and reducing the server’s ability to handle concurrent requests.

Improving file‑transfer performance therefore requires reducing both context‑switch frequency and memory‑copy count.

How does zero‑copy improve file‑transfer performance?

Zero‑copy eliminates most user‑space buffering by moving data directly between the kernel’s page cache and the network socket, merging the read and write system calls into a single operation. This cuts context switches roughly in half and reduces memory copies.

When the network card supports Scatter‑Gather DMA (SG‑DMA), the socket buffer copy can also be removed, leaving only two memory copies.

Zero‑copy’s kernel‑level function takes a file descriptor and a TCP socket, performing all data movement in kernel space, which not only reduces memory usage but also maximizes socket‑buffer utilization, further decreasing the number of required system calls.

PageCache – Disk cache

PageCache stores recently accessed disk data in memory, exploiting temporal locality. When the cache is full, the least‑recently‑used pages are evicted (LRU). Reads first check PageCache, dramatically speeding up disk access.

PageCache also pre‑fetches additional data beyond the requested range, so a small read (e.g., 0–32 KB) may cause the kernel to load the next 32 KB, reducing future read latency.

However, for very large files, PageCache can become a liability: large files occupy most of the cache, preventing small hot files from benefiting, and the extra copy into PageCache adds overhead. In high‑concurrency scenarios, it is better to bypass PageCache for large files.

Async I/O + Direct I/O

Asynchronous I/O separates the read request from the data retrieval, allowing the process to continue other work while the kernel fetches data. When the data is ready, the kernel notifies the process.

Async I/O does not use PageCache, which is a limitation. Direct I/O explicitly bypasses PageCache, sending data straight from disk to user buffers. It is useful when applications already implement their own caching (e.g., databases like MySQL) or when transferring large files that would otherwise pollute the cache.

Direct I/O sacrifices the kernel’s read‑ahead and request‑merging optimizations, so it should be used only when those benefits are outweighed by the cache‑related overhead.