Investigating an Nginx Reuseport Bug: How a Kernel Update Fixed Intermittent 403 Errors

The bug appeared on a machine where two nginx services (A and B) were co‑hosted on the same port 80 using the reuseport option. Service A listened on a specific virtual IP 6.6.6.6:80 reuseport , while service B listened on the wildcard address *:80 reuseport (i.e., INADDR_ANY). Occasionally, requests destined for A’s IP were delivered to B, resulting in 403 errors.

The root cause was identified after a restart of service B: its socket was placed ahead of A’s in the kernel’s listener hash table. Because B bound to the wildcard address, the kernel’s lookup (which at that time only hashed on port) would match B’s entry first, stealing the connection.

To understand why this happened, the article reviews networking fundamentals: the five‑tuple (protocol, local IP, local port, remote IP, remote port) uniquely identifies a connection; a socket is an implementation of the half‑tuple (protocol, local IP, local port). The bind() call can specify INADDR_ANY (0.0.0.0) to accept packets on any local IP.

The reuseport feature, available since Linux 3.9, allows multiple sockets to bind to the same IP/port pair. Its core implementation adds the SO_REUSEPORT socket option, modifies bind() to permit overlapping bindings, and changes connection acceptance to distribute load among listeners.

Prior to kernel 4.16‑rc, the kernel maintained a single hash table keyed by port only ( {port: socket_list} ). When several listeners shared a port, the kernel walked the list to find the matching socket, causing the ordering issue described above.

Starting with Linux 4.16‑rc+, the kernel keeps two hash tables: one keyed by IP+PORT and another by INADDR_ANY+PORT . Lookup first checks the IP+PORT table; only if no match is found does it fall back to the wildcard table. This change ensures that a request for 6.6.6.6:80 is matched to service A’s socket even after service B restarts, eliminating the bug.

The fix was achieved simply by upgrading the host kernel from version 4.14.81.bm.12‑amd64 to a 4.16‑rc+ release. No changes to nginx configuration were required, although the article notes that the underlying deployment (two services sharing the same IP/port) is suboptimal and could be better handled by merging configurations or using containers—though containers introduce proxy overhead for high‑performance components like nginx.