How Nginx Solves the Thundering Herd Problem: epoll, accept_mutex & SO_REUSEPORT

What is the thundering herd effect?

The thundering herd (or thundering herd) occurs when many processes or threads are blocked waiting for the same event; when the event occurs, all are awakened but only one can acquire the resource, forcing the others back to sleep and wasting CPU cycles.

Imagine a thunderstorm waking up a crowd of people, yet only one goes out to collect the laundry. Similarly, a single request wakes many processes, but only one actually handles it.

Causes & Issues

In high‑concurrency servers, multiple workers listen on the same socket. When a request arrives, all workers are notified, but only one can accept the connection, causing repeated wake‑sleep cycles and costly context switches.

Nginx Architecture

Nginx follows a master‑worker model: the master process handles configuration, signal processing, and listening socket setup, while worker processes handle the actual request processing.

The master only opens the listening sockets; workers inherit them and compete for incoming connections.

Nginx Uses epoll

Each worker creates its own epoll instance to monitor the shared listening socket, allowing efficient event‑driven I/O.

Master's Work

The master binds to the configured ports:

ngx_open_listening_sockets(ngx_cycle_t *cycle){
    ...
    for (i = 0; i < cycle->listening.nelts; i++) {
        ...
        if (bind(s, ls[i].sockaddr, ls[i].socklen) == -1) {
            if (listen(s, ls[i].backlog) == -1) {
                ...
            }
        }
    }
}

It then forks worker processes, copying the task structure so workers share the same resources.

Worker's Work

Workers load the epoll module (ngx_epoll_module.c) and create an epoll object:

ngx_epoll_init(ngx_cycle_t *cycle, ngx_msec_t timer){
    ngx_epoll_conf_t *epcf;
    epcf = ngx_event_get_conf(cycle->conf_ctx, ngx_epoll_module);
    if (ep == -1) {
        ep = epoll_create(cycle->connection_n / 2);
    }
    ...
}

Each worker’s epoll monitors the same listening socket, but only the worker that acquires the event will accept the connection.

Key Issue

When many workers are waiting on the same socket, the kernel may wake all of them, leading to the thundering herd problem.

How to Solve It

There are three main mitigation strategies:

accept_mutex (application‑level lock)

EPOLLEXCLUSIVE (kernel‑level flag)

SO_REUSEPORT (kernel‑level socket option)

accept_mutex

Workers compete for a mutex before calling accept(). The worker that holds the lock processes the request; others simply wait. This is simple but can become a bottleneck under heavy load.

EPOLLEXCLUSIVE

EPOLLEXCLUSIVE, added in Linux 4.5, reduces the probability of thundering herd by waking only one of the processes waiting on a shared file descriptor when an event occurs.

It does not eliminate the problem entirely; if the awakened worker is still busy, other workers may later find the socket already handled.

SO_REUSEPORT

Since Nginx 1.9.1, the reuseport socket option allows multiple workers to bind to the same port. The kernel performs load‑balancing at the accept stage, ensuring only one worker receives each new connection.

Configuration example:

http {
    server {
        listen 80 reuseport;
        server_name localhost;
        # ...
    }
}

Benchmarks show significant performance gains, but the kernel does not know whether a worker is busy, so a busy worker may still be selected, potentially causing latency spikes.

Conclusion

The article walks through the definition of the thundering herd effect, explains Nginx’s master‑worker design and epoll usage, and evaluates three mitigation techniques. While each method has trade‑offs, Nginx’s default model is sufficient for most real‑world scenarios.