Deep Dive into Network I/O: Principles, Socket Types, and epoll Multiplexing

1. Network I/O Overview

Network I/O (input/output) refers to the data exchange between a computer and the network, encompassing both inbound (input) and outbound (output) traffic. It is the foundation of all network interactions such as downloading files, streaming video, or sending emails.

2. Low‑Level Mechanisms

2.1 Hardware: NIC and Interrupts

The network interface card (NIC) receives electrical signals, converts them to bit streams via the PHY chip, encapsulates them into frames with the MAC chip, and transfers the data to memory using DMA. Once data reaches memory, the NIC raises an interrupt, prompting the CPU to run an interrupt handler that moves data to the socket’s receive buffer and wakes any blocked processes.

2.2 Process Scheduling and Blocking

When a process calls a blocking receive function (e.g., recv ) and no data is available, the process enters a blocked state, freeing the CPU for other ready tasks. The process is later awakened by the interrupt handler once data arrives.

2.3 I/O Models

Synchronous Blocking I/O (BIO) : The calling thread waits until the kernel finishes the read/write operation, consuming a thread per connection and causing scalability issues under high concurrency.

Synchronous Non‑Blocking I/O (NIO) : The call returns immediately with an error code (e.g., EWOULDBLOCK ) if data is not ready, allowing the application to perform other work and poll later.

Multiplexed I/O : System calls such as select , poll , and epoll let a single thread monitor many file descriptors. select and poll scan all descriptors linearly and have a fixed FD limit (1024 on many systems). epoll (Linux‑only) uses an event‑driven model with a ready‑list, eliminating the FD limit and reducing CPU overhead.

Asynchronous I/O (AIO) : The application issues an I/O request and continues execution; the kernel notifies completion via callbacks or signals, freeing the process from active waiting.

3. Socket Fundamentals

A socket is an endpoint for network communication, represented by a file descriptor. It abstracts network protocols as file‑like objects, allowing standard operations such as read and write .

3.1 Socket Types

SOCK_STREAM (TCP): Reliable, connection‑oriented byte stream with ordering and retransmission. Used by HTTP, file transfer, etc.

SOCK_DGRAM (UDP): Unreliable, message‑oriented datagrams without ordering or retransmission. Suitable for low‑latency scenarios like video chat.

SOCK_RAW : Direct access to lower‑level protocols (IP, ICMP) for tasks such as packet sniffing or custom protocol development.

4. epoll – High‑Performance I/O Multiplexing

4.1 Why epoll?

epoll has no hard limit on the number of file descriptors (limited only by system memory), unlike select / poll which cap at 1024 (see #define __FD_SETSIZE 1024 ). It also avoids linear scanning, handling only ready descriptors.

4.2 Core API

epoll_create creates an epoll instance and returns a file descriptor.

epoll_ctl adds, modifies, or removes descriptors from the epoll set (operations: EPOLL_CTL_ADD , EPOLL_CTL_MOD , EPOLL_CTL_DEL ).

epoll_wait blocks until one or more registered events become ready, returning the number of triggered events and filling a user‑allocated events array.

4.3 Trigger Modes

Level Trigger (LT) (default): The kernel reports an event as long as the condition holds (e.g., data remains in the receive buffer).

Edge Trigger (ET) : The kernel reports an event only when the condition changes (e.g., buffer transitions from empty to non‑empty). ET requires non‑blocking descriptors and careful handling to avoid missed notifications.

5. User Space vs. Kernel Space

User space processes run with limited privileges and cannot directly access hardware. System calls, hardware interrupts, and exceptions cause a controlled switch to kernel space, where privileged code performs I/O, memory management, and scheduling.

5.1 State Switch Triggers

System calls (e.g., open , socket ), hardware interrupts (e.g., NIC receiving a packet), and exceptions (e.g., page fault) all trigger a transition from user to kernel mode.

5.2 Switch Procedure

The CPU saves the user‑mode register context, copies arguments to kernel space, checks permissions, executes the kernel routine, copies results back, and finally restores the saved registers to resume user‑mode execution.

6. Conclusion

Understanding network I/O, socket APIs, and epoll’s event‑driven model equips backend developers to build scalable, high‑performance network services. Mastery of these concepts enables efficient handling of massive concurrent connections in e‑commerce platforms, real‑time communication apps, and online gaming servers.

cat /proc/sys/fs/file-max

#define __FD_SETSIZE    1024