Understanding Linux User/Kernel Space and I/O Models: From Blocking to Zero‑Copy

User Space and Kernel Space Concepts

In a 32‑bit Linux system the virtual address space is 4 GB; the upper 1 GB (0xC0000000‑0xFFFFFFFF) is reserved for the kernel (kernel space) while the lower 3 GB (0x00000000‑0xBFFFFFFF) is available to user processes (user space). Each process sees a full 4 GB virtual space but can only access its own user portion directly; kernel resources are accessed via system calls.

Linux Network I/O Models

Network I/O in Linux follows a request flow: a user process issues a system call (e.g., recvfrom), the kernel copies data from the device into an internal buffer, then copies the buffer to the user address space. The time spent waiting for data and the time spent copying define different I/O models.

Blocking I/O (Blocking I/O)

By default sockets are blocking. When a process calls recvfrom, the kernel first waits for enough data to arrive (stage 1). The process is blocked until the data is ready, then the kernel copies the data to user memory and returns, unblocking the process.

Non‑Blocking I/O (Non‑Blocking I/O)

Setting a socket to non‑blocking causes recvfrom to return immediately with an EWOULDBLOCK error if data is not yet ready. The process can perform other work and retry later; this pattern is often called polling.

I/O Multiplexing (I/O Multiplexing)

Select, poll, and epoll allow a single process to monitor many sockets. The kernel blocks the process in select (or poll) until at least one monitored descriptor becomes ready, then the process reads the ready data.

Signal‑Driven I/O (Signal Driven I/O)

The process installs a SIGIO handler and enables signal‑driven notifications on a socket. The process continues other work; when data becomes ready the kernel sends SIGIO, prompting the process to call recvfrom and copy the data.

Asynchronous I/O (Asynchronous I/O)

In async I/O the kernel is instructed to perform the operation and notifies the process only when the entire operation, including copying data to user space, is complete.

Zero‑Copy Techniques

Zero‑copy aims to eliminate unnecessary data copies between user space and kernel space, reducing CPU usage and memory bandwidth consumption during network transfers.

Cache I/O

Standard (cached) I/O copies data from the device into the page cache, then from the page cache into the user buffer, incurring two copies.

Zero‑Copy Classifications

Direct I/O: bypasses the page cache, transferring data directly between user buffers and the device.

DMA‑based transfers (e.g., mmap, sendfile, splice) avoid copying between kernel and user buffers.

Copy‑on‑write optimizations reduce the cost of copying between page cache and user buffers.

Using mmap to map a file into memory and then writing it directly to a socket eliminates one copy:

buf = mmap(diskfd, len);
write(sockfd, buf, len);

When using mmap, be aware of pitfalls such as SIGBUS on file truncation; solutions include installing a SIGBUS handler or using file lease locks (F_SETLEASE) to receive a real‑time signal before the file is truncated.

if (fcntl(diskfd, F_SETSIG, RT_SIGNAL_LEASE) == -1) {
    perror("kernel lease set signal");
    return -1;
}
/* set lease type */
if (fcntl(diskfd, F_SETLEASE, l_type)) {
    perror("kernel lease set type");
    return -1;
}