Investigating and Resolving a Massive Thread Leak in a Java Backend Application

In the afternoon, the author was alerted by a colleague about a performance problem in a neighboring department where the application had nearly 20,000 threads, with about 19,000 in a runnable state. The author downloaded a thread dump but could not pinpoint the cause because the stack traces lacked business code information.

Using SkyWalking, the author observed a steady increase in thread count without frequent GC activity, indicating that the threads were not being released. The thread dump revealed many threads named I/O dispatcher with identical stack traces, suggesting a common code path creating them.

Further inspection of the code uncovered a utility class HttpHelperAsyncClient that holds a CloseableHttpAsyncClient as a member variable. Each request creates a new client instance, but the client is closed via a static closeClient() method that operates on the shared member, leading to race conditions where one thread may close a client being used by another, causing leaked threads.

The author reproduced the issue by writing a simple infinite loop that repeatedly calls post("https://www.baidu.com", new Headers(), new HashMap<>(), 0) using the shared client, which only created a single main thread. Adding a multithreaded executor with many concurrent tasks caused the thread count to explode, confirming the leak.

To fix the problem, the author refactored the client creation to be a local variable within the execute() method, returning the client from an init() method and closing that local instance after use. This prevents the client object from escaping the method scope and eliminates the concurrency bug.

After deploying the changes, monitoring with SkyWalking showed the thread count stabilizing around 180 threads, a dramatic improvement over the previous growth to thousands of threads.

In summary, the article demonstrates a classic concurrency pitfall where shared resources (such as HTTP clients) are not safely managed, and it emphasizes the importance of keeping such resources as local variables to avoid thread leaks and improve application stability.