How Nginx Leverages epoll in a Multi‑Process Architecture

In a single‑process network model all socket operations—creation, bind, listen, epoll creation, event registration, and epoll_wait —are performed within one process. The article starts with a minimal C demo that shows a listening socket, an epoll object, and a loop that processes ready events.

1. Nginx Master Process Initialization

The master process is responsible for starting the program, reading configuration files, handling signals, and spawning worker processes. Its network duties are limited to creating a socket, binding it, and calling listen. The relevant code from src/core/nginx.c is:

int ngx_cdecl main(int argc, char * const *argv) {
    ngx_cycle_t *cycle, init_cycle;
    // 1.1 ngx_init_cycle opens listening sockets
    cycle = ngx_init_cycle(&init_cycle);
    // 1.2 start master event loop
    ngx_master_process_cycle(cycle);
}

Inside ngx_init_cycle (see src/core/ngx_cycle.c) the function ngx_open_listening_sockets iterates over the configured listen sockets, creates each socket, and performs bind and listen. Thus, the master only performs the bind/listen step.

2. Worker Process Responsibilities

After the master forks the configured number of workers, each worker runs ngx_worker_process_cycle (see src/os/unix/ngx_process_cycle.c). The worker performs three major tasks:

Creates an epoll object.

Registers the listening socket(s) with epoll_ctl.

Enters an event loop that calls epoll_wait and dispatches handlers.

Key snippets:

void ngx_worker_process_cycle(ngx_cycle_t *cycle, void *data) {
    // 2.2 initialize compiled modules
    ngx_worker_process_init(cycle, worker);
    // 2.3 event loop
    for (;;) {
        ngx_process_events_and_timers(cycle);
        ...
    }
}

The initialization function ngx_worker_process_init (same file) reads configuration, sets priorities, limits, user/group, CPU affinity, and then calls the init_process method of each loaded module.

static void ngx_worker_process_init(ngx_cycle_t *cycle, ngx_int_t worker) {
    // ... configuration handling ...
    for (i = 0; cycle->modules[i]; i++) {
        if (cycle->modules[i]->init_process) {
            if (cycle->modules[i]->init_process(cycle) == NGX_ERROR) {
                exit(2);
            }
        }
    }
    ...
}

The core event module ( src/event/ngx_event.c) provides ngx_event_process_init, which creates the epoll handle and registers each listening socket:

static ngx_int_t ngx_event_process_init(ngx_cycle_t *cycle) {
    // create epoll object via module actions
    module->actions.init(cycle, ngx_timer_resolution);
    // add each listening socket to epoll
    for (i = 0; i < cycle->listening.nelts; i++) {
        c = ngx_get_connection(ls[i].fd, cycle->log);
        rev->handler = ngx_event_accept;
        if (ngx_add_event(rev, NGX_READ_EVENT, 0) == NGX_ERROR) {
            return NGX_ERROR;
        }
    }
    return NGX_OK;
}

The ngx_add_event macro resolves to ngx_epoll_add_event, which simply wraps epoll_ctl(EPOLL_CTL_ADD) with the appropriate flags.

static ngx_int_t ngx_epoll_add_event(...){
    if (epoll_ctl(ep, EPOLL_CTL_ADD, c->fd, &ee) == -1) {
        ...
    }
    c->read->active = 1;
    c->write->active = 1;
    return NGX_OK;
}

2.1 Event Loop and Accept Handling

The worker’s main loop calls ngx_process_events_and_timers, which optionally acquires an accept mutex to avoid the thundering‑herd problem, then invokes ngx_process_events (mapped to ngx_epoll_process_events). This function performs epoll_wait and, for each ready event, either posts it for later processing or directly calls the registered handler.

void ngx_process_events_and_timers(ngx_cycle_t *cycle) {
    if (ngx_use_accept_mutex) {
        if (ngx_trylock_accept_mutex(cycle) == NGX_ERROR) {
            return;
        }
        if (ngx_accept_mutex_held) {
            flags |= NGX_POST_EVENTS;
        }
    }
    ngx_process_events(cycle, timer, flags);
}

When a listening socket becomes readable, the handler ngx_event_accept runs:

void ngx_event_accept(ngx_event_t *ev) {
    do {
        s = accept(lc->fd, &sa.sockaddr, &socklen);
        if (s) {
            c = ngx_get_connection(s, ev->log);
            if (ngx_add_conn(c) == NGX_ERROR) {
                ngx_close_accepted_connection(c);
                return;
            }
        }
    } while (ev->available);
}

ngx_get_connection

pulls a free ngx_connection_t from the cycle’s pool, and ngx_add_conn (for epoll, ngx_epoll_add_connection) registers the new client socket with the worker’s epoll instance.

ngx_connection_t *ngx_get_connection(ngx_socket_t s, ngx_log_t *log) {
    c = ngx_cycle->free_connections;
    c->fd = s;
    c->log = log;
    return c;
}

static ngx_int_t ngx_epoll_add_connection(ngx_connection_t *c) {
    struct epoll_event ee;
    ee.events = EPOLLIN|EPOLLOUT|EPOLLET|EPOLLRDHUP;
    ee.data.ptr = (void *)((uintptr_t) c | c->read->instance);
    epoll_ctl(ep, EPOLL_CTL_ADD, c->fd, &ee);
    c->read->active = 1;
    c->write->active = 1;
    return NGX_OK;
}

After the connection is added, the HTTP module’s ngx_http_init_connection sets the read handler to ngx_http_wait_request_handler and the write handler to ngx_http_empty_handler, preparing the request processing pipeline.

void ngx_http_init_connection(ngx_connection_t *c) {
    ngx_event_t *rev = c->read;
    rev->handler = ngx_http_wait_request_handler;
    c->write->handler = ngx_http_empty_handler;
}

3. Overall Flow Summary

The master process performs only socket creation, bind, and listen, then forks the configured number of workers. Each worker creates its own epoll object, registers the shared listening socket(s), and enters an event loop that waits on epoll_wait. When a client connects, the accept handler creates a connection object, adds it to epoll, and hands it to the HTTP module for request processing. The design avoids the thundering‑herd problem by optionally using an accept mutex.

In a multi‑worker deployment each worker holds its own epoll instance but monitors the same set of listening sockets; only one worker will acquire the accept mutex (or be the first to succeed without the mutex) and thus handle a particular client connection, preventing the thundering‑herd issue.