Analysis and Optimization of HttpClient Connection Management in High‑Concurrency Scenarios

HttpClient is the most widely used HTTP tool for Java developers, simplifying connection management and improving reuse efficiency. However, under high concurrency and large payloads, network fluctuations can cause connection‑pool problems that manifest as service outages.

Problem symptoms observed on a specific date included circuit‑breaker activation for several HTTP interfaces, massive RequestAbortedException logs, and a rapid increase of connections in the CLOSE_WAIT state, approaching the pool's maximum of 250 connections.

Initial diagnosis suggested TCP connection management issues on the affected machines. After restarting the VMs, the problem temporarily disappeared, but re‑appeared on other machines, indicating a service‑level root cause.

Temporary mitigation involved increasing the pool size to 500 and restarting the machines, which stopped the immediate outage.

Root‑cause analysis revealed that the abnormal CLOSE_WAIT accumulation was not due to missing stream closures—code already called IOUtils.closeQuietly(is) and IOUtils.closeQuietly(httpResponse) —but stemmed from an interface returning large (~500 KB) responses that triggered client‑side ACK failures, leaving connections in CLOSE_WAIT .

Key diagnostic commands used:

netstat -ant|awk '/^tcp/ {++S[$NF]} END {for(a in S) print (a,S[a])}'

netstat -tulnap|grep CLOSE_WAIT

The investigation then turned to HttpClient's internal connection‑pool implementation. Important classes include PoolingHttpClientConnectionManager , CPool , CPoolEntry , HttpClientBuilder , MainClientExec , and ConnectionHolder . The pool maintains four containers: free , leased , pending , and available , each governing the lifecycle of connections.

Connection creation follows these steps:

Attempt to fetch a valid connection from available and move it to leased .

If none are available, check whether the sum of available and leased exceeds maxPerRoute ; if so, release excess connections.

If the total is still below limits, create a new connection or recycle the oldest one from available .

If all attempts fail, wait for connectionRequestTimeout .

When a request finishes, long‑living (reusable) connections are returned to available with an expiry timestamp, while short‑living connections are closed and returned to free . By default, connections without an explicit expiry are considered permanent ( Long.MAX_VALUE ), which explains why CLOSE_WAIT sockets persisted.

Key configuration recommendations derived from the analysis:

Set an explicit keep‑alive timeout, e.g., new PoolingHttpClientConnectionManager(60, TimeUnit.SECONDS) , to bound connection lifetimes.

Enable periodic idle‑connection eviction, such as evictIdleConnections(5, TimeUnit.SECONDS) , to clean up stale sockets.

Adjust maxTotal and maxPerRoute carefully; consider separate HttpClient instances for distinct traffic patterns.

Avoid closing the entire HttpResponse object after each request; only close the data stream to allow connection reuse.

Implementing these changes eliminated the continuous growth of CLOSE_WAIT connections and restored stable service performance.

Finally, the article notes two major shortcomings of HttpClient: lack of built‑in connection‑state monitoring and a synchronous lock‑based connection acquisition that can become a bottleneck under high concurrency.