Unlocking Multi‑Process Server Speed: How Linux REUSEPORT Works

Problem addressed by SO_REUSEPORT

Historically a service could listen on a fixed port (e.g., Nginx on 80, MySQL on 3306). As web traffic grew after 2010, the single‑bind model became a bottleneck for high‑concurrency servers. Two classic multi‑process designs were used:

A dispatcher process accepts new connections and forwards them to worker processes, incurring an extra context switch and creating a dispatcher bottleneck.

Multiple workers share one listening socket and accept directly (as Nginx does), but they must serialize access with a lock, causing lock contention.

Both approaches suffer performance penalties under massive load.

SO_REUSEPORT introduction

Linux 3.9 (2013) introduced the SO_REUSEPORT socket option, allowing several user‑space processes to bind and listen on the same port while the kernel performs load balancing.

Kernel commit references: https://github.com/torvalds/linux/commit/da5e36308d9f7151845018369148201a5d28b46d and https://github.com/torvalds/linux/commit/055dc21a1d1d219608cd4baac7d0683fb2cbbe8

Enabling the option

setsockopt(fd, SOL_SOCKET, SO_REUSEPORT, ...);

This call sets the kernel socket field sk_reuseport to 1.

Bind‑time handling

During inet_bind the kernel calls inet_csk_get_port. If both the existing socket and the new socket have SO_REUSEPORT enabled and belong to the same user namespace, the bind succeeds; otherwise a bind conflict occurs.

if (tb->fastreuseport > 0 && sk->sk_reuseport && uid_eq(tb->fastuid, uid)) {
    goto success; // bind succeeds
} else {
    // bind conflict
}

The uid_eq check ensures that only sockets owned by the same UID can share the port, preventing cross‑user hijacking.

Accept‑time load balancing

When a client connects, the kernel looks up all listening sockets with the same port using __inet_lookup_listener. It computes a score for each socket based on factors such as destination IP match and socket family. The socket with the highest score wins; ties are broken pseudo‑randomly.

score = compute_score(sk, net, hnum, daddr, dif);
if (score > hiscore) {
    result = sk;
    hiscore = score;
    reuseport = sk->sk_reuseport;
    if (reuseport) {
        phash = inet_ehashfn(...);
        matches = 1;
    }
} else if (score == hiscore && reuseport) {
    matches++;
    if (((u64)phash * matches) >> 32 == 0)
        result = sk;
    phash = next_pseudo_random32(phash);
}

The scoring logic prefers sockets bound to the exact destination IP (score 4) over wildcard bindings (score 2). This explains why a process bound to 10.0.0.2:6000 receives connections to that address, while a process bound to 0.0.0.0:6000 handles the rest.

Practical demonstration

Minimal server implementation that enables SO_REUSEPORT is available at: https://github.com/yanfeizhang/coder-kung-fu/blob/main/tests/network/test08/server.c

Running multiple instances on the same port

$ ./test-server 0.0.0.0 6000
Start server on 0.0.0.0:6000 successed, pid is 23179

Running the same command in several terminals shows that all instances start successfully, confirming that the port is shared.

Verifying kernel load balancing

With three identical servers listening on 0.0.0.0:6000, a client that opens many connections receives roughly equal numbers of connections on each server, demonstrating random distribution.

Server 0.0.0.0 6000 (23179) accept success:15
Server 0.0.0.0 6000 (23177) accept success:25
Server 0.0.0.0 6000 (23185) accept success:20

Priority matching test

On a machine with two IPs (10.0.0.2 and 10.0.0.3), start:

A: ./test-server 10.0.0.2 6000
B: ./test-server 0.0.0.0 6000

Connecting to 10.0.0.2 hits process A (score 4). Connecting to 10.0.0.3 falls back to process B (score 2).

Cross‑user security check

If a server is started by user alice with SO_REUSEPORT, a second user (e.g., root ) attempting to bind the same port receives “Bind Failed!” because the UID check blocks cross‑user reuse.

Summary

Before Linux 3.9 a port could be bound by only one socket, limiting multi‑process server scalability. The REUSEPORT feature introduced in 3.9 lets multiple processes bind the same port, and the kernel performs random or score‑based load balancing, eliminating dispatcher bottlenecks and lock contention.

In Nginx 1.9.1+ the feature can be enabled with a single line:

server {
    listen 80 reuseport;
    ...
}