Understanding and Solving the Linux Thundering Herd Problem

What is the thundering herd effect?

The thundering herd (also called the thunderstorm effect) occurs when multiple processes or threads are blocked waiting for the same event; when the event occurs, all of them are awakened, but only one can acquire the event and handle it, while the others go back to sleep, causing wasted CPU cycles and context switches.

An analogy is a group of pigeons startled by a single grain of seed: only one pigeon gets the seed while the rest return to sleep.

What does the thundering herd consume?

1) Excessive, useless scheduling and context‑switch overhead for the awakened processes/threads, degrading overall system performance.

2) Additional locking required to ensure only one entity handles the event, further increasing system load.

The true nature of the thundering herd

From both process and thread perspectives, the effect can be observed in three common scenarios: accept() calls, epoll_wait() , and thread condition broadcasts.

1) accept() thundering herd

Scenario: a parent creates a listening socket, forks several child processes, and each child blocks on accept(). When a new connection arrives, all children are awakened, but only one succeeds in accepting the connection.

#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <sys/wait.h>
#include <string.h>
#include <netinet/in.h>
#include <unistd.h>

#define PROCESS_NUM 10
int main() {
    int fd = socket(PF_INET, SOCK_STREAM, 0);
    int connfd;
    int pid;
    char sendbuff[1024];
    struct sockaddr_in serveraddr;
    serveraddr.sin_family = AF_INET;
    serveraddr.sin_addr.s_addr = htonl(INADDR_ANY);
    serveraddr.sin_port = htons(1234);
    bind(fd, (struct sockaddr *)&serveraddr, sizeof(serveraddr));
    listen(fd, 1024);
    for (int i = 0; i < PROCESS_NUM; ++i) {
        pid = fork();
        if (pid == 0) {
            while (1) {
                connfd = accept(fd, NULL, NULL);
                snprintf(sendbuff, sizeof(sendbuff), "process %d accepted
", getpid());
                send(connfd, sendbuff, strlen(sendbuff)+1, 0);
                printf("process %d accept success
", getpid());
                close(connfd);
            }
        }
    }
    wait(0);
    return 0;
}

Running this program and connecting with telnet shows that only one process prints a success message, confirming that the kernel wakes only one process for accept() in modern Linux (2.6+).

2) epoll_wait() thundering herd

When multiple processes block on epoll_wait() for the same listening socket, a new connection can wake all waiting processes. Only one will successfully accept(), the others receive EAGAIN.

#include <stdio.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <unistd.h>
#include <sys/epoll.h>
#include <netdb.h>
#include <stdlib.h>
#include <fcntl.h>
#include <sys/wait.h>
#include <errno.h>

#define PROCESS_NUM 10
#define MAXEVENTS 64

int sock_create_bind(char *port) {
    int sock_fd = socket(AF_INET, SOCK_STREAM, 0);
    struct sockaddr_in serveraddr;
    serveraddr.sin_family = AF_INET;
    serveraddr.sin_port = htons(atoi(port));
    serveraddr.sin_addr.s_addr = htonl(INADDR_ANY);
    bind(sock_fd, (struct sockaddr *)&serveraddr, sizeof(serveraddr));
    return sock_fd;
}

int make_nonblocking(int fd) {
    int val = fcntl(fd, F_GETFL);
    val |= O_NONBLOCK;
    if (fcntl(fd, F_SETFL, val) < 0) {
        perror("fcntl set");
        return -1;
    }
    return 0;
}

int main(int argc, char *argv[]) {
    if (argc < 2) { printf("usage: %s [port]
", argv[1]); exit(1); }
    int sock_fd = sock_create_bind(argv[1]);
    make_nonblocking(sock_fd);
    listen(sock_fd, SOMAXCONN);
    int epoll_fd = epoll_create(MAXEVENTS);
    struct epoll_event event, *events = calloc(MAXEVENTS, sizeof(event));
    event.data.fd = sock_fd; event.events = EPOLLIN;
    epoll_ctl(epoll_fd, EPOLL_CTL_ADD, sock_fd, &event);
    for (int i = 0; i < PROCESS_NUM; ++i) {
        int pid = fork();
        if (pid == 0) {
            while (1) {
                int num = epoll_wait(epoll_fd, events, MAXEVENTS, -1);
                printf("process %d returned from epoll_wait
", getpid());
                sleep(2);
                for (int j = 0; j < num; ++j) {
                    if (events[j].data.fd == sock_fd) {
                        if (accept(sock_fd, NULL, NULL) < 0)
                            printf("process %d accept failed! %s
", getpid(), strerror(errno));
                        else
                            printf("process %d accept successful!
", getpid());
                    }
                }
            }
        }
    }
    wait(0);
    free(events);
    close(sock_fd);
    return 0;
}

Strace shows that only one process gets the connection (the others receive EAGAIN), confirming a partial thundering herd. Adding a sleep(2) after epoll_wait() forces all processes to be awakened, demonstrating the effect.

3) Thread thundering herd

When a condition variable is broadcast (e.g., pthread_cond_broadcast()), all waiting threads are awakened, but only one obtains the resource. The usual remedy is to protect the critical section with a mutex.

printf("Initial red‑packet: <count:%d amount:%d.%02d>
", item.number, item.total/100, item.total%100);
pthread_cond_broadcast(&temp.cond); // wake all threads
pthread_mutex_unlock(&temp.mutex);
sleep(1);

How to solve the thundering herd?

Common approaches include:

1) Nginx accept mutex

Nginx workers use an accept mutex (configurable via accept_mutex) so that only one worker calls accept() at a time. The mutex is non‑blocking; if a worker cannot obtain it, it backs off, reducing wake‑ups.

void ngx_process_events_and_timers(ngx_cycle_t *cycle) {
    if (ngx_use_accept_mutex) {
        if (ngx_accept_disabled > 0) {
            ngx_accept_disabled--;
        } else {
            if (ngx_trylock_accept_mutex(cycle) == NGX_ERROR) return;
            if (ngx_accept_mutex_held) {
                flags |= NGX_POST_EVENTS;
            } else {
                if (timer == NGX_TIMER_INFINITE || timer > ngx_accept_mutex_delay) {
                    timer = ngx_accept_mutex_delay;
                }
            }
        }
    }
    ngx_process_events(cycle, timer, flags);
    if (ngx_posted_accept_events) {
        ngx_event_process_posted(cycle, &ngx_posted_accept_events);
    }
    if (ngx_accept_mutex_held) {
        ngx_shmtx_unlock(&ngx_accept_mutex);
    }
    if (delta) ngx_event_expire_timers();
}

2) SO_REUSEPORT

Since Linux 3.9, the SO_REUSEPORT socket option allows multiple processes or threads to bind to the same IP:port. The kernel then load‑balances incoming connections among the sockets, eliminating lock contention on accept().

With SO_REUSEPORT, each worker has its own listening socket, no mutex is needed, and the kernel performs per‑connection hashing to select the target worker.

3) Comparison of traditional multi‑process/thread servers vs. SO_REUSEPORT

Traditional models suffer from a single listener bottleneck, lock contention on the listening socket, cache‑line bouncing, and uneven load distribution. SO_REUSEPORT removes these issues by providing lock‑free parallel accept and kernel‑level load balancing.

Conclusion

The thundering herd effect can degrade server performance in accept, epoll, and thread‑based designs. Modern Linux kernels mitigate the accept case, but epoll and thread wake‑ups may still exhibit the problem. Using accept mutexes, SO_REUSEPORT, and careful locking are effective ways to eliminate unnecessary wake‑ups and improve scalability.