How Nginx Leverages epoll in a Multi‑Process Architecture

Single‑process network model

In a single‑process model all network operations—creating the listening socket, bind, listen, creating an epoll instance, registering events with epoll_ctl, and waiting for events with epoll_wait —are performed inside one process. A minimal demonstration looks like this:

int main(){
    // create listening socket
    int lfd = socket(AF_INET, SOCK_STREAM, 0);
    bind(lfd, ...);
    listen(lfd, ...);

    // create epoll and register the listening socket
    int efd = epoll_create1(0);
    struct epoll_event ev = { .events = EPOLLIN, .data.fd = lfd };
    epoll_ctl(efd, EPOLL_CTL_ADD, lfd, &ev);

    // event loop
    for(;;){
        struct epoll_event events[64];
        int nready = epoll_wait(efd, events, 64, -1);
        for(int i = 0; i < nready; ++i){
            if(events[i].data.fd == lfd){               // new connection
                int fd = accept(lfd, NULL, NULL);
                struct epoll_event evc = { .events = EPOLLIN|EPOLLET, .data.fd = fd };
                epoll_ctl(efd, EPOLL_CTL_ADD, fd, &evc);
            } else {
                // read/write or close the client socket
                ...
            }
        }
    }
}

Redis 5.0 and earlier use a very similar loop; because the workload is almost entirely memory‑bound, the single‑process design can still achieve tens of thousands of queries per second.

Why a multi‑process model is needed

A single process cannot fully exploit multi‑core CPUs. Production services therefore adopt a multi‑process architecture, which raises coordination questions such as which process performs listen, which accepts new connections, how connections are distributed, and whether separate processes are required for heavy computation.

1. Nginx master process initialization

The master process starts the program, reads the configuration, handles signals, and manages worker processes. Its network duties are limited to creating the listening socket and performing bind and listen:

// src/core/nginx.c
int ngx_cdecl main(int argc, char * const *argv) {
    ngx_cycle_t *cycle, init_cycle;
    cycle = ngx_init_cycle(&init_cycle);   // opens listening sockets
    ngx_master_process_cycle(cycle);       // forks workers and enters master loop
}

ngx_init_cycle

eventually calls ngx_open_listening_sockets, which iterates over all listen entries, creates a socket, and invokes bind and listen for each:

// src/core/ngx_cycle.c
if (ngx_open_listening_sockets(cycle) != NGX_OK) {
    goto failed;
}

// src/core/ngx_connection.c
ls = cycle->listening.elts;
for (i = 0; i < cycle->listening.nelts; i++) {
    s = ngx_socket(ls[i].sockaddr->sa_family, ls[i].type, 0);
    bind(s, ls[i].sockaddr, ls[i].socklen);
    listen(s, ls[i].backlog);
    ls[i].listen = 1;
    ls[i].fd = s;
}

Thus, bind and listen are performed only by the master process.

1.1 Master main loop

ngx_master_process_cycle

starts the configured number of workers via fork and then waits for signals (quit, reload, restart, etc.).

// src/os/unix/ngx_process_cycle.c
ngx_start_worker_processes(cycle, ccf->worker_processes, NGX_PROCESS_RESPAWN);
for (;;) {
    // handle signals
    ...
}

Each worker is created with ngx_spawn_process, which simply calls fork and runs ngx_worker_process_cycle in the child.

2. Worker process responsibilities

Every worker performs the bulk of the network work:

Initializes compiled modules.

Creates an epoll object.

Registers all listening sockets with epoll.

Enters the event loop, calling epoll_wait to receive connection and I/O events.

// src/os/unix/ngx_process_cycle.c
void ngx_worker_process_cycle(ngx_cycle_t *cycle, void *data) {
    ngx_worker_process_init(cycle, worker);
    for (;;) {
        ngx_process_events_and_timers(cycle);
        ...
    }
}

The initialization routine reads core configuration, sets process priority, file‑descriptor limits, user/group IDs, CPU affinity, and then invokes the init_process hook of every loaded module.

2.1 Event module and epoll integration

The core event module registers a set of function pointers (actions) that abstract the underlying mechanism. For the epoll module these actions map to concrete implementations such as ngx_epoll_add_event and ngx_epoll_process_events:

// src/event/modules/ngx_epoll_module.c
static ngx_event_module_t ngx_epoll_module_ctx = {
    &epoll_name,
    ngx_epoll_create_conf,
    ngx_epoll_init_conf,
    {
        ngx_epoll_add_event,
        ngx_epoll_del_event,
        ngx_epoll_add_event,
        ngx_epoll_del_event,
        ngx_epoll_add_connection,
        ngx_epoll_del_connection,
        ngx_epoll_notify,
        ngx_epoll_process_events,
        ngx_epoll_init,
        ngx_epoll_done
    }
};

During ngx_event_process_init the worker creates the epoll handle and adds each listening socket:

// src/event/ngx_event.c
static ngx_int_t ngx_event_process_init(ngx_cycle_t *cycle) {
    // create epoll object via the event module's init action
    for (m = 0; cycle->modules[m]; m++) {
        if (cycle->modules[m]->type != NGX_EVENT_MODULE) continue;
        module->actions.init(cycle, ngx_timer_resolution);
        break;
    }
    // register listening sockets
    ls = cycle->listening.elts;
    for (i = 0; i < cycle->listening.nelts; i++) {
        c = ngx_get_connection(ls[i].fd, cycle->log);
        rev->handler = ngx_event_accept;
        if (ngx_add_event(rev, NGX_READ_EVENT, 0) == NGX_ERROR) {
            return NGX_ERROR;
        }
    }
    return NGX_OK;
}

2.2 Event loop

The function ngx_process_events_and_timers optionally acquires an accept‑mutex to avoid the thundering‑herd problem, then calls the module‑specific process_events implementation, which for epoll is a thin wrapper around epoll_wait:

// src/event/modules/ngx_epoll_module.c
static ngx_int_t ngx_epoll_process_events(ngx_cycle_t *cycle, ...) {
    int events = epoll_wait(ep, event_list, (int) nevents, timer);
    for (i = 0; i < events; i++) {
        if (flags & NGX_POST_EVENTS) {
            ngx_post_event(rev, queue);
        } else {
            rev->handler(rev);
        }
    }
    return NGX_OK;
}

3. Accepting a client connection

When a listening socket becomes readable, ngx_event_accept runs:

// src/event/ngx_event_accept.c
void ngx_event_accept(ngx_event_t *ev) {
    do {
        int s = accept(ev->fd, &sa, &socklen);
        if (s != -1) {
            ngx_connection_t *c = ngx_get_connection(s, ev->log);
            if (ngx_add_conn(c) == NGX_ERROR) {
                ngx_close_accepted_connection(c);
                return;
            }
        }
    } while (ev->available);
}

The newly accepted socket is wrapped in a ngx_connection_t, added to the epoll set via ngx_epoll_add_connection, and its read handler is set to ngx_http_wait_request_handler (for HTTP):

// src/event/modules/ngx_epoll_module.c
static ngx_int_t ngx_epoll_add_connection(ngx_connection_t *c) {
    struct epoll_event ee;
    ee.events = EPOLLIN|EPOLLOUT|EPOLLET|EPOLLRDHUP;
    ee.data.ptr = (void *)((uintptr_t) c | c->read->instance);
    epoll_ctl(ep, EPOLL_CTL_ADD, c->fd, &ee);
    c->read->active = 1;
    c->write->active = 1;
    return NGX_OK;
}

// src/http/ngx_http_request.c
void ngx_http_init_connection(ngx_connection_t *c) {
    ngx_event_t *rev = c->read;
    rev->handler = ngx_http_wait_request_handler;
    c->write->handler = ngx_http_empty_handler;
}

4. Summary of the workflow

The master process creates the listening socket and performs bind / listen.

The master forks the configured number of workers.

Each worker creates its own epoll instance, registers the shared listening socket, and enters an event loop.

When epoll_wait reports the listening socket as readable, the worker runs ngx_event_accept, accepts the client, obtains a ngx_connection_t, adds it to epoll, and hands it to the HTTP module.

Subsequent I/O on the client socket is processed by the same worker’s epoll loop.

In a multi‑worker deployment every worker holds its own epoll handle and monitors all listening sockets, but only one worker actually processes a given client connection, eliminating duplicate handling and allowing the server to scale across CPU cores.