Why HttpClient Connections Stay in CLOSE_WAIT and How to Fix Them

HttpClient is a widely used Java HTTP tool that simplifies connection handling and improves reuse efficiency, but under high concurrency and large payloads combined with network instability, connection management can degrade, leading to excessive CLOSE_WAIT sockets.

1. Problem Symptoms

On a specific day the monitoring system reported three issues: (1) Hystrix circuit breakers opened for several HTTP endpoints, (2) the affected machines showed many connections in the CLOSE_WAIT state, and (3) logs contained numerous

org.apache.http.impl.execchain.RequestAbortedException: Request aborted

entries.

2. Initial Diagnosis

Running

netstat -ant | awk '/^tcp/ {++S[$NF]} END {for(a in S) print (a,S[a])}'

on the problematic host revealed around 200 CLOSE_WAIT connections, close to the pool's max size of 250. Restarting the VM cleared the issue temporarily.

3. Temporary Mitigation

After the incident spread to other machines, a job was added to monitor TCP state distribution. The connection pool size was increased to 500 and the VM was rebooted, which stopped the immediate outage.

4. Root‑Cause Analysis

Further investigation showed that the problem was not due to missing close() calls—code already closed streams and responses—but rather that certain large‑payload API calls (≈500 KB) caused the server to close the socket without the client sending the final ACK, leaving the client side in CLOSE_WAIT.

Packet captures confirmed the missing ACK handshake, matching the suspect API.

5. Deep Dive into HttpClient Internals

The analysis focused on HttpClient’s long‑connection management. Key classes include PoolingHttpClientConnectionManager , CPool , CPoolEntry , HttpClientBuilder , MainClientExec , and ConnectionHolder . HttpClient maintains four containers for connections: free , leased , pending , and available .

Connection acquisition follows these steps:

Try to fetch a reusable connection from available . If valid (not expired), move it to leased and return it.

If none, check whether free + leased exceeds maxPerRoute. If so, release excess connections.

If total connections are below limits, create a new one; otherwise, evict the oldest available connection.

If still unavailable, wait up to connectionRequestTimeout for a connection.

Connection release works as follows:

For reusable (keep‑alive) connections, move them from leased to the head of available and set an expiry time.

For non‑reusable connections, close them and return them to free .

Wake any threads waiting for a connection.

By default, if no explicit expiry is set, HttpClient treats the connection as permanent ( Long.MAX_VALUE). The PoolingHttpClientConnectionManager constructor that accepts a validityDeadline can adjust this behavior.

6. Final Conclusions and Recommendations

Only close the data stream, not the entire connection, unless the target is truly random.

Configure maxTotal and maxPerRoute carefully; consider using separate HttpClient instances for isolated traffic.

Set a reasonable keep‑alive expiry, e.g., new PoolingHttpClientConnectionManager(60, TimeUnit.SECONDS), to avoid permanent connections.

Enable idle‑connection eviction, such as evictIdleConnections(5, TimeUnit.SECONDS), to clean up stale sockets under high concurrency.

These adjustments eliminated the growth of CLOSE_WAIT sockets and restored stable service performance.