Why TIME_WAIT Connections Accumulate and How to Fix Them

Problem Description

When a server experiences a burst of short‑lived connections (e.g., under high concurrency or as an Nginx reverse proxy), a large number of TCP sockets enter the TIME_WAIT state. Each TIME_WAIT socket occupies a local port (max 65535). If many sockets stay in TIME_WAIT, new connections may fail with address already in use: connect.

Observations

Some TIME_WAIT sockets are reclaimed quickly, while new ones keep appearing.

In extreme cases, the number of TIME_WAIT sockets can grow dramatically.

Think: What business impact does a massive amount of TIME_WAIT connections have?

Technical Background

The active closer of a TCP connection enters TIME_WAIT after sending the final ACK of the four‑handshake. The socket remains in this state for 2 × MSL (Maximum Segment Lifetime), typically 4 minutes (MSL ≈ 2 minutes). This delay ensures that delayed packets are discarded and that the remote side can retransmit its final FIN if the ACK was lost.

Key facts:

Each TIME_WAIT socket consumes one local port.

Port numbers are 16‑bit, so the theoretical limit is 65 535.

When many sockets stay in TIME_WAIT, new connections may be rejected.

How to Inspect TCP States

$ netstat -n | awk '/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}'

Typical output on a busy server:

ESTABLISHED 1154
TIME_WAIT 1645

On macOS you can list TIME_WAIT sockets directly:

$ netstat -nat | grep TIME_WAIT
$ netstat -nat | grep -E "TIME_WAIT|Local Address"

Root Causes

大量短连接导致频繁主动关闭。

HTTP 请求的 Connection: close 头部让服务器主动关闭 TCP。

TCP 四次挥手的设计要求在 TIME_WAIT 保持 2 × MSL（约 4 分钟）。

Mitigation Strategies

Client side: use Connection: keep-alive so the client keeps the socket open for multiple requests.

Server side:

Enable socket reuse (e.g., SO_REUSEADDR or SO_REUSEPORT) to allow a new socket to bind to a port still in TIME_WAIT.

Reduce the TIME_WAIT timeout to 1 MSL (≈ 2 minutes) if the OS permits.

Key Takeaways

TIME_WAIT is entered by the side that actively closes the connection.

Default TIME_WAIT duration is 2 × MSL (≈ 4 minutes).

Each TIME_WAIT socket blocks its local port until the timeout expires.

The total number of TCP ports is limited to 65 535.

Excessive TIME_WAIT sockets can cause address already in use errors.

Appendix A – Querying TCP States on macOS

$ netstat -nat | grep TIME_WAIT
$ netstat -nat | grep -E "TIME_WAIT|Local Address"

Appendix B – MSL (Maximum Segment Lifetime)

MSL is the maximum time a TCP segment can exist in the network before being discarded. RFC 793 defines MSL as 2 minutes, but many systems use shorter defaults (30 s, 1 min, etc.).

Appendix C – 2 MSL (TIME_WAIT) Details

After the active closer sends the final ACK, it enters TIME_WAIT.

The socket must remain in this state for 2 × MSL to handle possible retransmitted FINs.

During TIME_WAIT, both endpoints cannot reuse the port.

Setting SO_REUSEADDR can bypass the wait in many OS implementations.

Appendix D – TCP Handshake Diagrams

Three‑way handshake (connection establishment) and four‑way termination (connection release) are illustrated below.