Understanding Nginx Multi‑Process Network Architecture and Epoll Usage

The article begins by describing a single‑process network model where all socket creation, binding, listening, and epoll operations occur in one process, and shows a simple C example that demonstrates socket setup, epoll creation, and an event loop.

int main() {
  // listen
  lfd = socket(AF_INET, SOCK_STREAM, 0);
  bind(lfd, ...);
  listen(lfd, ...);

  // create epoll and add listen socket
  efd = epoll_create(...);
  epoll_ctl(efd, EPOLL_CTL_ADD, lfd, ...);

  for (;;) {
    size_t nready = epoll_wait(efd, ep, ...);
    for (int i = 0; i < nready; ++i) {
      if (ep[i].data.fd == lfd) {
        fd = accept(lfd, ...);
        epoll_ctl(efd, EPOLL_CTL_ADD, fd, ...);
      } else {
        // handle read/write
      }
    }
  }
}

It then points out the limitations of a single‑process model and introduces the two‑process design used by Nginx: a Master process and multiple Worker processes.

1. Nginx Master Process Initialization

The Master process only creates sockets, performs bind and listen , and then forks the configured number of Worker processes. The relevant source snippets show the ngx_init_cycle function opening listening sockets and the ngx_master_process_cycle function spawning workers.

// src/core/nginx.c
int ngx_cdecl main(int argc, char const *argv[]) {
  ngx_cycle_t *cycle, init_cycle;
  cycle = ngx_init_cycle(&init_cycle);
  ngx_master_process_cycle(cycle);
}

Listening sockets are stored in cycle->listening , and the Master does not perform any epoll‑related calls.

2. Worker Process Handling

Each Worker process creates its own epoll instance, registers the listening sockets with epoll_ctl , and enters an event loop that calls epoll_wait . The code for initializing the epoll object and adding connections is shown in ngx_epoll_init and ngx_epoll_add_connection .

// src/event/modules/ngx_epoll_module.c
static ngx_int_t ngx_epoll_init(ngx_cycle_t *cycle, ngx_msec_t timer) {
  ep = epoll_create(cycle->connection_n / 2);
  ngx_event_actions = ngx_epoll_module_ctx.actions;
}

static ngx_int_t ngx_epoll_add_connection(ngx_connection_t *c) {
  struct epoll_event ee;
  ee.events = EPOLLIN|EPOLLOUT|EPOLLET|EPOLLRDHUP;
  ee.data.ptr = (void *)((uintptr_t)c | c->read->instance);
  epoll_ctl(ep, EPOLL_CTL_ADD, c->fd, &ee);
  c->read->active = 1;
  c->write->active = 1;
  return NGX_OK;
}

When a listening socket becomes readable, the Worker invokes ngx_event_accept , which calls accept , obtains a free ngx_connection_t via ngx_get_connection , and registers the new client socket with epoll.

// src/event/ngx_event_accept.c
void ngx_event_accept(ngx_event_t *ev) {
  do {
    s = accept(lc->fd, &sa.sockaddr, &socklen);
    if (s) {
      c = ngx_get_connection(s, ev->log);
      if (ngx_add_conn(c) == NGX_ERROR) {
        ngx_close_accepted_connection(c);
        return;
      }
    }
  } while (ev->available);
}

The newly accepted connection is then initialized by the HTTP module (or other modules) with callbacks such as ngx_http_wait_request_handler , allowing the Worker to process client data.

3. Summary of the Full Flow

The Master creates and binds listening sockets, then forks Workers.

Each Worker creates its own epoll object, registers all listening sockets, and starts an event loop.

When a client connects, epoll notifies a Worker, which accepts the connection and adds the client socket to epoll.

Subsequent read/write events on the client socket are handled by the module‑specific handlers.

The article concludes with a discussion of how Nginx avoids the “thundering herd” problem using accept mutexes, and invites readers to think about the coordination between multiple Workers listening on the same sockets.