How ReusePort Solves the Thundering Herd Problem in Workerman

Thundering Herd Problem

When multiple processes or threads wait on the same event (e.g., accept or epoll), the kernel wakes all of them when the event occurs. Only one process actually acquires the event; the others return to sleep, causing unnecessary context switches and high CPU usage.

Thundering Herd in Workerman

Workerman uses a master/worker model built with pcntl_fork(). Each worker creates a listening socket via stream_socket_server(). Because the forked workers inherit the same socket descriptor, a new connection wakes every worker, all of which compete for the same accept call. Only one succeeds, while the rest perform a wasted wake‑up and go back to waiting – the classic thundering herd effect.

SO_REUSEPORT (ReusePort)

The SO_REUSEPORT socket option (available since Linux 3.9) allows multiple processes or threads to bind and listen on the same IP:PORT pair. The kernel then distributes incoming connections across the bound sockets using a hash‑based load‑balancing algorithm, eliminating the need for all workers to be awakened simultaneously.

How ReusePort Works

Each process gets its own independent listen‑socket queue, avoiding contention for a single accept queue.

The kernel balances new connections among the sockets, providing near‑equal distribution without extra user‑space coordination.

For TCP, the kernel creates a distinct listener socket per thread, improving load distribution compared with a single shared accept thread.

For UDP, the same principle reduces packet‑receive contention among processes.

Enabling ReusePort in Workerman

Set the worker property $worker->reusePort = true. The relevant part of Workerman’s Worker class is:

protected function listen()
{
    // SO_REUSEPORT.
    if ($this->reusePort) {
        \stream_context_set_option($this->_context, 'socket', 'so_reuseport', 1);
    }
    // Create an Internet or Unix domain server socket.
    $this->_mainSocket = \stream_socket_server($local_socket, $errno, $errmsg, $flags, $this->_context);
    ...
}

When reusePort is false, the master process creates the socket before forking, so all workers share the same descriptor. When true, each worker creates its own socket after forking, allowing the kernel to apply the load‑balancing provided by SO_REUSEPORT.

Behavior Without ReusePort

Requests tend to concentrate on a few workers, leaving others idle. This concentration increases CPU usage due to repeated wake‑ups and context switches.

Behavior With ReusePort

Connections are distributed almost evenly across all workers, eliminating the thundering herd overhead and improving overall throughput.

Common Misconceptions

Adding more worker processes does not always increase performance. For simple, CPU‑bound workloads, the optimal number of workers matches the number of CPU cores. Over‑provisioning adds context‑switch overhead and can degrade throughput. For workloads involving I/O or databases, a higher worker count (e.g., 3‑6 × CPU cores) may be beneficial.

Implementation Details

When reusePort is false, the master calls listen() before pcntl_fork(), so the created socket descriptor is inherited by all child workers:

protected function initWorkers()
{
    ...
    // Listen.
    if (! $worker->reusePort) {
        $worker->listen();
    }
    ...
}

When reusePort is true, the master does not call listen() before forking. After each worker is forked, it invokes listen() itself, creating an independent socket with SO_REUSEPORT set:

/**
 * Fork one worker process (Linux).
 */
protected static function forkOneWorkerForLinux($worker)
{
    $pid = pcntl_fork();
    if ($pid === 0) {
        $worker->run();
    }
    ...
}

public function run()
{
    $this->listen();
    ...
}

References

https://www.workerman.net/doc/workerman/faq/requests-concentrated-in-certain-processes.html

https://www.jianshu.com/p/97cc8c52d47a

https://www.workerman.net/doc/workerman/worker/listen.html

https://www.workerman.net/doc/workerman/worker/reuse-port.html