Zero‑Copy in Linux: Boost Server Performance with mmap, sendfile & splice

Introduction

When a server program needs to send a file over a socket, the naive approach reads the file into a user‑space buffer and then writes that buffer to the socket, causing multiple data copies and context switches between user and kernel space.

The following diagram illustrates the four copies involved in the traditional read/write path:

What is Zero‑Copy?

Zero‑copy aims to eliminate unnecessary data copies by allowing the kernel to transfer data directly between buffers without involving the CPU for each byte. This reduces CPU load and improves overall system efficiency.

Using mmap

Replace the read call with mmap to map the file directly into the process address space, avoiding the user‑space copy.

buf = mmap(diskfd, len);
write(sockfd, buf, len);

The diagram below shows the data flow with mmap:

While mmap reduces one copy, it introduces pitfalls such as SIGBUS when the underlying file is truncated. Common mitigations include installing a SIGBUS handler or using file lease locks:

if (fcntl(diskfd, F_SETSIG, RT_SIGNAL_LEASE) == -1) {
    perror("kernel lease set signal");
    return -1;
}
/* l_type can be F_RDLCK/F_WRLCK for locking, F_UNLCK for unlocking */
if (fcntl(diskfd, F_SETLEASE, l_type)) {
    perror("kernel lease set type");
    return -1;
}

Using sendfile

Since Linux 2.1, sendfile transfers data directly from a file descriptor to a socket descriptor, staying entirely in kernel space.

#include <sys/sendfile.h>
ssize_t sendfile(int out_fd, int in_fd, off_t *offset, size_t count);

Diagram of the sendfile process:

sendfile

eliminates the user‑space copy and reduces context switches, but it only works when the input file is mmap‑able and the output is a socket.

Using splice

Linux 2.6.17 introduced splice to move data between two file descriptors without copying to user space. At least one descriptor must be a pipe.

#define _GNU_SOURCE
#include <fcntl.h>
ssize_t splice(int fd_in, loff_t *off_in, int fd_out, loff_t *off_out, size_t len, unsigned int flags);

Key flag values:

SPLICE_F_MOVE : Hint to move data instead of copying (may fall back to copy).

SPLICE_F_NONBLOCK : Do not block the operation.

SPLICE_F_MORE : More data will follow in subsequent calls.

Because splice relies on the pipe buffer mechanism, it is limited to scenarios where a pipe can be used.

Other Techniques

Linux also employs copy‑on‑write (COW) to avoid copying data until a write occurs, and flags like O_DIRECT can bypass the page cache for direct I/O. Advanced methods such as fbufs exist but are beyond the scope of this article.

Author: 卡巴拉的树 Source: https://www.jianshu.com/p/fad3339e3448