How SO_REUSEPORT Boosts NGINX Performance and Enables Seamless Rolling Upgrades

NGINX 1.9.1 introduced a new feature: the SO_REUSEPORT socket option, available on newer operating systems such as DragonFly BSD and Linux kernels 3.9 or later. This option permits multiple sockets to listen on the same IP‑port combination, with the kernel performing load‑balancing of incoming connections.

(For NGINX Plus customers, this feature will appear in version 7 released at the end of the year.)

SO_REUSEPORT has many practical applications. Other services can use it to implement simple rolling upgrades, and for NGINX it can reduce lock contention in certain scenarios, thereby improving performance.

When the option is effective, a single listening socket notifies worker processes of incoming connections, and each worker thread attempts to acquire the connection.

With SO_REUSEPORT enabled, multiple socket listeners are bound to each IP and port, and each worker process can be assigned one. The kernel decides which effective socket listener (implicitly, which worker) receives a connection, reducing mutex contention when acquiring new connections and improving performance on multi‑core systems. However, if a worker becomes blocked, the blockage can affect not only that worker but also the kernel’s scheduling of connections to other workers.

Configuring Shared Sockets

To make the SO_REUSEPORT socket option work, add the reuseport parameter directly to the listen directive in HTTP or TCP (stream) blocks, as shown below:

http {
    server {
        listen 80 reuseport;
        server_name localhost;
        ...
    }
}

stream {
    server {
        listen 12345 reuseport;
        ...
    }
}

After adding the reuseport parameter, the accept_mutex setting becomes ineffective for those sockets, because the mutex is redundant with reuseport. For ports that do not use reuseport, accept_mutex remains valuable.

Benchmarking Reuseport Performance

I ran wrk benchmarks on a 36‑core AWS instance with four NGINX workers, keeping client and NGINX on the same host and having NGINX return a simple "OK" string. Three configurations were compared: default (equivalent to accept_mutex on), accept_mutex off, and reuseport. The reuseport configuration achieved two to three times more requests per second, with lower latency and latency standard deviation.

In a second test, client and NGINX were on separate machines and NGINX served an HTML file. Reuseport again reduced latency, with the standard deviation dropping to about one‑tenth of the default case. Load was evenly distributed among workers with reuseport, whereas the default configuration caused some workers to handle a disproportionately high share of the load.

Note that reuseport cannot be used in the mail context (e.g., for email), because email traffic does not match this scenario. We recommend testing before large‑scale deployment. For additional NGINX performance tuning tips, see Konstantin Pavlov’s talk at the nginx2014 conference.

Acknowledgments

Thanks to Sepherosa Ziehau and Yingqi Lu for contributing a socket‑option solution to the NGINX project. The NGINX team integrated their ideas into what we believe is an ideal solution.