Why Did My Dubbo Service Crash? A Deep Dive into Thread Dumps, GC, and Memory Leaks

Monitoring Log Analysis

During the deployment of a new feature, the third machine’s CPU spiked to 300% and Dubbo active threads jumped over 1000, forcing an immediate rollback. Log inspection revealed that the filterMission method, a high‑QPS interface (≈200 million calls per day), occasionally took more than 30 seconds, but its implementation contains no heavy computation, large data handling, or lock contention, so it was ruled out as the direct cause.

Code Review

A thorough code review found no synchronized blocks, complex calculations, or differences from the previous version, confirming that the recent code change was not responsible for the failure.

Thread Dump Analysis

The team captured a thread‑dump to investigate further. Below is a simplified illustration of Java thread states:

Key states explained:

NEW : thread object just created.

RUNNABLE : ready to run, awaiting CPU.

WAITING : waiting on Object.wait(), Thread.join() or LockSupport.park().

TIMED_WAITING : waiting with a timeout (e.g., Thread.sleep()).

BLOCKED : waiting to acquire a monitor lock.

TERMINATED : thread has finished execution.

Analyzing jstack Logs

Many Dubbo threads were in WAITING because they were parked while the thread‑pool queue was empty. The thread‑pool creates new threads up to corePoolSize when requests arrive; otherwise, workers block on the queue.

Another group of threads was also in WAITING , blocked on a SynchronousQueue transfer, indicating they were waiting for tasks from the ScheduledThreadPoolExecutor.

Numerous threads were in BLOCKED while trying to acquire the monitor of RollingRandomAccessFileManager, a Log4j component responsible for rolling log files.

"DubboServerHandler-172.24.16.78:33180-thread-1220" #1700 daemon prio=5 os_prio=0 tid=0x00007f3394988800 nid=0x4aae waiting on condition [0x00007f32d75c0000]
    java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    ...

These logs showed that threads were parked waiting for tasks or resources, leading to request timeouts.

Memory Leak Discovery

Further investigation revealed continuous memory growth without reduction, and the JVM was performing frequent full GC cycles. The root cause was a pagination query that omitted LIMIT and OFFSET, causing the entire table (≈3 million rows) to be loaded and processed in memory, preventing GC from reclaiming space.

Thread‑Pool Misconfiguration

The Dubbo thread‑pool was configured with a maximum of 1500 threads. Under high concurrency, this large pool caused excessive context switching, high CPU usage, and severe lock contention, further degrading throughput.

Conclusion

The incident was ultimately caused by a memory leak from an unbounded pagination query, which triggered continuous full GC, CPU spikes, and a surge in active Dubbo threads. Additionally, an oversized thread‑pool amplified lock contention. The post documents the debugging steps to help avoid similar pitfalls and emphasizes the importance of monitoring, log analysis, and proper resource configuration in production environments.