Understanding and Handling Excessive TIME_WAIT TCP Connections in High‑Concurrency Scenarios

Problem description : In simulated high‑concurrency environments a massive amount of TCP connections enter the TIME_WAIT state, which can later disappear after being reclaimed. While this is normal for short‑lived connections, the sheer volume may affect services, especially when Nginx acts as a reverse proxy.

Impact : Each TIME_WAIT entry consumes a local port (max 65535). When many ports are in TIME_WAIT , new connections can fail with address already in use: connect errors.

Analysis : The abundance of TIME_WAIT stems from a high rate of short connections, often caused by HTTP requests with the Connection: close header, which forces the server to actively close the socket. TCP’s four‑handshake closes the connection and keeps the socket in TIME_WAIT for twice the Maximum Segment Lifetime (MSL), typically 2 × 2 minutes, to ensure ACK retransmission and to discard delayed packets.

Key points about TIME_WAIT :

The side that actively closes the connection enters TIME_WAIT .

The state lasts for 2 MSL (about 4 minutes by default).

Ports in TIME_WAIT cannot be reused until the timer expires.

Excessive TIME_WAIT can trigger the "address already in use" error.

Solutions :

Enable socket reuse (e.g., set SO_REUSEADDR ) so that TIME_WAIT sockets can be reclaimed earlier.

Reduce the TIME_WAIT timeout to 1 MSL (≈2 minutes) if the OS permits.

Configure clients to use Connection: keep-alive to keep connections alive longer and reduce the rate of short connections.

On the server side, avoid actively closing connections unless necessary.

Practical commands for inspecting TCP states:

// Count various connection states
$ netstat -n | awk '/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}'
ESTABLISHED 1154
TIME_WAIT 1645

On macOS, you can list TIME_WAIT connections directly:

$ netstat -nat | grep TIME_WAIT

Appendix A – Query TCP connection state (macOS) :

// Mac query example
$ netstat -nat | grep -E "TIME_WAIT|Local Address"
Proto Recv-Q Send-Q Local Address Foreign Address (state)
... (output omitted) ...

Appendix B – MSL (Maximum Segment Lifetime) : Defined as the maximum time a TCP segment can exist in the network; RFC 793 sets it to 2 minutes, though implementations may use 30 seconds or 1 minute.

Appendix C – TCP three‑way handshake and four‑way termination are referenced for deeper understanding of connection establishment and teardown.

Overall, the article clarifies why TIME_WAIT appears, its necessity for reliable TCP termination, and how to mitigate its impact on high‑traffic services.