How to Tame Linux TIME_WAIT: Optimize tcp_tw_reuse, tcp_tw_recycle & fin_timeout

Linux TIME_WAIT related parameters:

net.ipv4.tcp_tw_reuse = 0    # 0 = disabled, 1 = enable reuse of TIME-WAIT sockets
net.ipv4.tcp_tw_recycle = 0  # 0 = disabled, 1 = enable fast recycle of TIME-WAIT sockets
net.ipv4.tcp_fin_timeout = 60  # seconds a socket stays in FIN-WAIT-2 (can be reduced)

Note: Unlike Windows, Linux cannot modify the MSL value; tcp_fin_timeout controls FIN‑WAIT‑2, not the 2MSL timeout.

tcp_tw_reuse vs. SO_REUSEADDR: they are completely different. Both tw_reuse and tw_recycle work only when TCP timestamps are enabled (default on).

Client side

1. Enabling tcp_tw_reuse helps clients avoid port exhaustion caused by many TIME_WAIT sockets; do not enable tcp_tw_recycle simultaneously.

2. tcp_tw_reuse allows a client to recycle a socket within about 1 second, enabling roughly 60 000 requests per second on a single machine.

3. In internal‑network stress tests where the client does not accept inbound connections, tcp_tw_recycle can provide a slight benefit.

4. Applications can also design the server to close connections proactively.

Server side

1. Enabling tcp_tw_reuse has no effect on the server.

2. In production environments, tcp_tw_recycle should not be enabled because it may cause connection failures, especially behind NAT or load balancers that strip timestamps.

3. High TIME_WAIT counts on servers are usually not a problem; Linux already optimizes memory usage for TIME_WAIT sockets.

4. The maximum number of TIME_WAIT buckets can be increased with tcp_max_tw_buckets = 262144, which is sufficient for modern hardware.

Example of a busy HTTP short‑connection service handling 10 000 requests per second shows:

ss -s
Total: 259 (kernel 494)
TCP:   262419 (estab 113, closed 262143, orphaned 156, synrecv 0, timewait 262143/0), ports 80

Memory usage (slabtop) indicates ~70 MB used by tw_sock_TCP with a high bucket count.

System logs may repeatedly show “TCP: time wait bucket table overflow”, suggesting the bucket limit should be raised.

Principle analysis

1. MSL origin : After the side that initiates closure sends the final FIN, it waits 2 MSL (≈4 minutes) to ensure delayed FINs do not interfere with new connections.

2. Reuse & recycle : They rely on TCP timestamps; a newer timestamp indicates a new connection, preventing old FINs from closing it.

3. Reuse : Allows client and server to discard out‑of‑order FINs based on timestamp progression.

4. Recycle : On the server side, NAT environments may break timestamp monotonicity, causing the server to reject connections that appear non‑incremental.

Further reading:

“Coping with the TCP TIME‑WAIT state on busy Linux servers”

“tcp_tw_recycle parameter troubleshooting”

“tcp_tw_reuse parameter troubleshooting”