Master Blocking vs Non‑Blocking I/O: From select to epoll and Reactor Patterns

Blocking vs Non‑Blocking I/O

Blocking I/O suspends the calling thread until the kernel completes the operation, wasting resources when many connections are idle. Non‑blocking I/O returns immediately, allowing the program to continue processing other tasks.

Problems of the traditional one‑thread‑per‑connection model

High concurrency requires a large number of threads, exhausting memory and CPU.

If a thread has no data to read, it blocks on read, wasting the thread.

I/O Multiplexing

select

Monitors a set of file descriptors for readability, writability or errors.

int select(int maxfdp, fd_set *readset, fd_set *writeset, fd_set *exceptset, struct timeval *timeout);

maxfdp

is the highest descriptor value plus one. The three fd_set arguments specify which descriptors to watch.

poll

Uses an array of struct pollfd to avoid the descriptor limit of select.

int poll(struct pollfd *fds, nfds_t nfds, int timeout);

struct pollfd { int fd; short events; short revents; };

Typical event flags: POLLIN – data available for reading. POLLOUT – ready for writing.

epoll (Linux)

Scales to thousands of descriptors by keeping a kernel‑side event list. Supports level‑triggered and edge‑triggered modes.

int epoll_create(int size);

int epoll_ctl(int epfd, int op, int fd, struct epoll_event *event);

int epoll_wait(int epfd, struct epoll_event *events, int maxevents, int timeout);

epoll_wait

returns the number of ready events, 0 on timeout, or –1 on error.

Solving the C10K Problem

Handling 10 000 concurrent connections requires careful management of file descriptors, memory buffers and CPU resources.

Raise the descriptor limit (e.g., fs.file-max in /etc/sysctl.conf).

Allocate sufficient send/receive buffers; 128 KB per connection can exceed 1 GB for 10 000 connections.

Use an I/O strategy that avoids one‑thread‑per‑connection.

Process‑per‑connection (fork)

Each client is handled by a separate process. High overhead and zombie‑process management via a SIGCHLD handler are required.

Thread‑per‑connection

Each connection gets its own thread. Threads share the address space, reducing memory usage compared to processes, but large thread counts still cause context‑switch overhead.

Thread pool + non‑blocking I/O

A fixed pool of worker threads processes tasks taken from a connection queue. The main thread uses epoll (or poll) to detect ready sockets, reads data, and dispatches work to the pool.

Reactor pattern

An event loop waits for I/O readiness (via epoll) and invokes callbacks. Variants:

Single‑reactor, single‑thread.

Single‑reactor, multi‑thread (worker threads handle callbacks).

Master‑slave reactors: a master accepts new connections and distributes them to multiple sub‑reactors, typically one per CPU core.

Proactor vs Reactor

Reactor notifies when an I/O operation *can* be performed (e.g., socket becomes readable). Proactor (e.g., Windows IOCP) notifies when an I/O operation *has completed*, and the application processes the completed event.

Practical Code Samples

Minimal select -based server (Linux, IPv4, port 10001):

#include <stdio.h>
#include <string.h>
#include <arpa/inet.h>
#include <netinet/in.h>
#include <sys/socket.h>
#include <sys/select.h>

int main() {
    int listenfd = socket(AF_INET, SOCK_STREAM, 0);
    struct sockaddr_in servaddr;
    memset(&servaddr, 0, sizeof(servaddr));
    servaddr.sin_family = AF_INET;
    servaddr.sin_port = htons(10001);
    servaddr.sin_addr.s_addr = htonl(INADDR_ANY);
    bind(listenfd, (struct sockaddr *)&servaddr, sizeof(servaddr));
    listen(listenfd, 100);
    fd_set allset, rset;
    FD_ZERO(&allset);
    FD_SET(listenfd, &allset);
    int maxfd = listenfd;
    for (;;) {
        rset = allset;
        int nready = select(maxfd + 1, &rset, NULL, NULL, NULL);
        if (FD_ISSET(listenfd, &rset)) {
            // accept new connection ...
        }
        // handle other ready descriptors ...
    }
    return 0;
}

Minimal epoll -based server (Linux, IPv4, port 10001):

#include <stdio.h>
#include <string.h>
#include <arpa/inet.h>
#include <netinet/in.h>
#include <sys/socket.h>
#include <sys/epoll.h>

#define MAX_EVENTS 1024

int main() {
    int listenfd = socket(AF_INET, SOCK_STREAM, 0);
    struct sockaddr_in servaddr;
    memset(&servaddr, 0, sizeof(servaddr));
    servaddr.sin_family = AF_INET;
    servaddr.sin_port = htons(10001);
    servaddr.sin_addr.s_addr = htonl(INADDR_ANY);
    bind(listenfd, (struct sockaddr *)&servaddr, sizeof(servaddr));
    listen(listenfd, 100);

    int epfd = epoll_create(1);
    struct epoll_event ev, events[MAX_EVENTS];
    ev.events = EPOLLIN;
    ev.data.fd = listenfd;
    epoll_ctl(epfd, EPOLL_CTL_ADD, listenfd, &ev);

    for (;;) {
        int n = epoll_wait(epfd, events, MAX_EVENTS, -1);
        for (int i = 0; i < n; ++i) {
            if (events[i].data.fd == listenfd) {
                // accept new connection and add to epoll set
            } else if (events[i].events & EPOLLIN) {
                // read data from socket
            }
        }
    }
    return 0;
}

Combining non‑blocking sockets, epoll, and a thread pool enables a server to handle tens of thousands of simultaneous connections without the overhead of one thread per client.