Unlocking Linux SO_REUSEPORT: How Multiple Processes Can Share the Same Port

Problem solved by SO_REUSEPORT

Before Linux kernel 3.9 a TCP/UDP port could be bound by only one socket. Any additional bind attempt returned “Address already in use”, which limited scalability for high‑traffic servers.

Background

The limitation motivated the addition of the SO_REUSEPORT socket option in Linux 3.9 (2013). The design discussion is recorded in the kernel commits:

https://github.com/torvalds/linux/commit/da5e36308d9f7151845018369148201a5d28b46d

https://github.com/torvalds/linux/commit/055dc21a1d1d219608cd4baac7d0683fb2cbbe8

Enabling SO_REUSEPORT

In user space the option is enabled with a single setsockopt call on the listening socket:

setsockopt(fd, SOL_SOCKET, SO_REUSEPORT, &val, sizeof(val));

The kernel sets the internal flag sk_reuseport to 1 (see net/core/sock.c).

// file: net/core/sock.c
int sock_setsockopt(struct socket *sock, int level, int optname,
                    char __user *optval, unsigned int optlen)
{
    ...
    switch (optname) {
        ...
        case SO_REUSEPORT:
            sk->sk_reuseport = valbool;
            break;
        ...
    }
    ...
}

Bind handling with REUSEPORT

When bind() is called, the kernel executes inet_csk_get_port() (file net/ipv4/inet_connection_sock.c). It walks the hash table of existing sockets. If a socket with the same port exists in the same network namespace, the bind succeeds only when both sockets have SO_REUSEPORT enabled; otherwise a conflict is reported.

// file: net/ipv4/inet_connection_sock.c
int inet_csk_get_port(struct sock *sk, unsigned short snum)
{
    ...
    /* locate the bucket for the requested port */
    head = &hashinfo->bhash[inet_bhashfn(net, snum, hashinfo->bhash_size)];
    inet_bind_bucket_for_each(tb, &head->chain) {
        if (net_eq(ib_net(tb), net) && tb->port == snum) {
            /* potential conflict – check REUSEPORT flags */
            if ((tb->fastreuseport > 0 && sk->sk_reuseport &&
                 uid_eq(tb->fastuid, uid)) ||
                (tb->fastreuse > 0 && sk->sk_reuse &&
                 sk->sk_state != TCP_LISTEN)) {
                goto success;   /* bind allowed */
            } else {
                /* bind conflict */
                return -EADDRINUSE;
            }
        }
    }
    ...
success:
    /* allocate the port to the socket */
    ...
}

The additional check uid_eq(tb->fastuid, uid) ensures that only sockets owned by the same user can share the port, preventing cross‑user hijacking.

Listener selection (accept handling)

When a new connection arrives, the kernel searches the listening_hash for sockets bound to the destination address/port. Each candidate receives a score computed by compute_score(). The socket with the highest score is chosen; ties are broken by a pseudo‑random function, providing lock‑free load balancing.

// file: net/ipv4/inet_hashtables.c
struct sock *__inet_lookup_listener(struct net *net,
                                     struct inet_hashinfo *hashinfo,
                                     __be32 saddr, __be16 sport,
                                     __be32 daddr, unsigned short hnum,
                                     int dif)
{
    struct inet_listen_hashbucket *ilb = &hashinfo->listening_hash[hash];
    struct sock *result = NULL;
    int hiscore = 0;
    u32 phash = 0;
    int matches = 0;
    sk_nulls_for_each_rcu(sk, node, &ilb->head) {
        int score = compute_score(sk, net, hnum, daddr, dif);
        if (score > hiscore) {
            result = sk;
            hiscore = score;
            if (sk->sk_reuseport) {
                phash = inet_ehashfn(net, daddr, hnum, saddr, sport);
                matches = 1;
            }
        } else if (score == hiscore && sk->sk_reuseport) {
            matches++;
            if (((u64)phash * matches) >> 32 == 0)
                result = sk;
            phash = next_pseudo_random32(phash);
        }
    }
    return result;
}

The scoring function prefers sockets bound to a specific IP address over the wildcard 0.0.0.0 and gives a higher weight to IPv4 sockets when both families are present.

// file: net/ipv4/inet_hashtables.c
static inline int compute_score(struct sock *sk, ...)
{
    int score = -1;
    struct inet_sock *inet = inet_sk(sk);
    if (net_eq(sock_net(sk), net) && inet->inet_num == hnum &&
        !ipv6_only_sock(sk)) {
        __be32 rcv_saddr = inet->inet_rcv_saddr;
        score = (sk->sk_family == PF_INET) ? 2 : 1;   // IPv4 gets 2, IPv6 gets 1
        if (rcv_saddr) {               // socket bound to a concrete address
            if (rcv_saddr != daddr)
                return -1;           // address mismatch – not a candidate
            score += 4;                // extra weight for exact address match
        }
        /* additional heuristics omitted for brevity */
    }
    return score;
}

Practical demonstration

A minimal C server that enables SO_REUSEPORT can be compiled and launched in several terminals. All instances bind to the same port (e.g., 6000) and start successfully:

$ ./test-server 0.0.0.0 6000
Start server on 0.0.0.0:6000 successed, pid is 23179
$ ./test-server 0.0.0.0 6000
Start server on 0.0.0.0:6000 successed, pid is 23177

When a client opens many connections, the accept() counters of each process are roughly equal, showing kernel‑level load balancing:

Server 0.0.0.0:6000 (23179) accept success:15
Server 0.0.0.0:6000 (23177) accept success:25
Server 0.0.0.0:6000 (23185) accept success:20
Server 0.0.0.0:6000 (23181) accept success:19
Server 0.0.0.0:6000 (23183) accept success:21

Priority rules verification

Assume a machine with two IP addresses, 10.0.0.2 and 10.0.0.3. Three processes are started:

A: ./test-server 10.0.0.2 6000
B: ./test-server 0.0.0.0 6000
C: ./test-server 127.0.0.1 6000

Connections to 10.0.0.2 are handled by process A because its socket receives a score of 6 (IPv4 + exact address). Connections to 10.0.0.3 are not matched by A; the wildcard socket B receives a score of 2 and therefore handles the traffic. Process C only matches local loopback connections.

$ telnet 10.0.0.2 6000   # hits process A
$ telnet 10.0.0.3 6000   # hits process B

Cross‑user safety check

If a non‑root user binds a port with SO_REUSEPORT, a different user (including root) cannot bind the same port, because the kernel requires matching UID when fastreuseport is used.

# ./test-server 0.0.0.0 6000   # started by user alice – succeeds
# su - root
# ./test-server 0.0.0.0 6000   # fails: Bind Failed!

Conclusion

Prior to kernel 3.9 only a single socket could own a port, forcing either a single‑process accept loop or a dispatcher‑worker model that suffered from context‑switch overhead and lock contention. The SO_REUSEPORT feature introduced in Linux 3.9 allows multiple processes to bind the same port. The kernel performs lock‑free, hash‑based load balancing by scoring candidate listening sockets, thereby improving scalability for high‑concurrency servers.