Boost NGINX Performance with SO_REUSEPORT: Configuration and Benchmark Results

NGINX 1.9.1 adds a new feature: the SO_REUSEPORT socket option, available on recent BSD and Linux kernels (3.9+). This option lets multiple sockets listen on the same IP‑port pair, letting the kernel balance incoming connections among them, which can reduce lock contention and improve performance.

When SO_REUSEPORT is enabled, each worker process can obtain a separate listening socket. The kernel implicitly decides which socket receives each connection, decreasing mutex contention between workers. However, if a worker blocks, the blockage can affect the kernel’s connection‑distribution decisions.

Setting a Shared Socket

To enable SO_REUSEPORT for HTTP or TCP stream listeners, add the reuseport parameter to the listen directive, as shown in the example image below.

When reuseport is used, the accept_mutex setting becomes redundant and is ignored; for ports without reuseport, accept_mutex remains useful. By default accept_mutex is on, and two related directives are provided:

1) accept_mutex

The mutex is a load‑balancing lock that serializes request handling among workers. When enabled, workers take turns processing requests; disabling it lets all workers accept new connections, which may waste resources under low load.

2) lock_file

When the load‑balancing lock is active, a lock file must be specified. NGINX uses this lock mechanism to implement accept_mutex and serialize access to shared memory.

Reuseport Benchmark Tests

Using a 36‑core AWS instance and the wrk tool, three NGINX configurations were compared: default (equivalent to accept_mutex on), accept_mutex off, and reuseport. Results show that reuseport achieved two to three times higher requests per second, with lower latency and reduced latency variance.

Another test with client and NGINX on separate machines, serving an HTML file, confirmed similar latency reductions; the standard deviation dropped to about one‑tenth. With reuseport, load was evenly distributed among workers, whereas the default configuration caused some workers to receive a disproportionate share of traffic, and disabling accept_mutex resulted in all workers being heavily loaded.

These benchmarks involve high connection rates with lightweight request handling. In real‑world traffic that matches this pattern, reuseport can provide substantial performance gains. Note that reuseport cannot be used in the mail context. Users are encouraged to test in their own environments before large‑scale deployment. For additional performance‑tuning tips, see Konstantin Pavlov’s talk at the nginx2014 conference.

Original source: http://www.ywnds.com/?p=4543