Analyzing Intermittent Timeouts and RST Packets in Apache HttpClient Calls

At the end of last year a colleague shared an intermittent timeout case where Department A called an HTTPS interface of Department B; logs showed no server‑side timeout, but tcpdump on the client revealed many RST packets and a large number of connections in CLOSE_WAIT state.

The root cause was identified in the client code, which created a new DefaultHttpClient for each request, set connection and socket timeouts, executed the request, and returned the response without properly releasing the connection back to the pool, leading to connection leakage.

Because each request instantiated its own HttpClient, the underlying connection pool filled up; when Nginx closed idle connections after its keep‑alive timeout, the client, still holding leaked sockets, responded with RST packets.

The team fixed the issue by adding httpGet.setHeader("Connection", "close") , which forces the connection to close after each request, but this disables keep‑alive and defeats the purpose of the HttpClient pool. A better solution is to reuse a single HttpClient instance across calls.

Further analysis explored why RST packets were observed instead of the expected FIN. TLS close_notify messages are sent before TCP shutdown; if the client closes the socket while unread data remains in the receive buffer, TCP must send an RST according to RFC 1122. Experiments using a custom SSLSocketFactory to obtain the raw socket confirmed that closing the raw socket still produced an RST, and that reading the receive buffer completely before closing eliminated the RST.

Additional tests showed that when the client sends a proper close_notify followed by a FIN after the buffer is drained, the server may still send an RST in some directions, highlighting subtle differences in server implementations (e.g., Nginx vs. Netty) and the impact of the Connection: close header.

The article concludes with several thought questions about the effect of the Connection: close header on RST generation, why Nginx does not emit RST in certain cases, and why a Netty‑based client behaves differently.