Diagnosing Dubbo Thread Pool Exhaustion with Arthas: A Step‑by‑Step Guide

Preface

This article is the second in the Arthas series.

Dubbo Thread‑Pool Exhaustion Overview

Dubbo uses a default thread pool of 200 threads on the provider side; when all threads are busy the client receives an exception such as:

Caused by: java.util.concurrent.ExecutionException: org.apache.dubbo.remoting.RemotingException: Server side (192.168.1.101,20880) threadpool is exhausted ...

The server also logs a WARN message: [DUBBO] Thread pool is EXHAUSTED! Unreasonable timeout settings causing requests to wait indefinitely.

Excessive request volume.

Full GC or slow processing.

Blocking I/O (database, Redis, network).

…

The root cause is usually a business‑level problem that exhausts Dubbo’s thread resources.

Investigate business exceptions first.

Adjustment Suggestions

Increase dubbo.provider.threads (default 200) to a suitable value, e.g., 700.

Adjust dubbo.consumer.actives to limit call rate.

Apply client‑side rate limiting.

Scale the provider.

Note: Dubbo does not yet support per‑service isolated thread pools.

Reproducing the Issue

Configure Provider Thread Pool Size

dubbo.protocol.threads=10

Setting a small pool makes the problem easier to reproduce.

Simulate Server Blocking

@Service(version = "1.0.0")
public class DemoServiceImpl implements DemoService {
    @Override
    public String sayHello(String name) {
        sleep();
        return "Hello " + name;
    }
    private void sleep() {
        try {
            Thread.sleep(5000);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
    }
}

Client Multi‑Thread Access

for (int i = 0; i < 20; i++) {
    new Thread(() -> {
        while (true) {
            Thread.sleep(1000);
            demoService.sayHello("Provider");
        }
    }).start();
}

Running the client and server shows the exhaustion scenario.

Arthas dashboard Command

$ dashboard

The dashboard displays system information; focus on the THREAD panel. Columns include ID, NAME, GROUP, PRIORITY, STATE, CPU%, TIME, INTERRUPTED, DAEMON.

Because the service method sleeps, all Dubbo threads are in TIMED_WAITING, causing subsequent requests to fail.

Arthas thread Command

After filtering Dubbo threads with: dashboard | grep "DubboServerHandler" use thread to inspect individual threads.

Show top N busy threads: thread -n 3 Show all threads: thread Show threads blocking others: thread -b Show threads in a specific state: thread --state TIMED_WAITING Show stack of a thread:

thread 46

Thread State Overview

NEW : thread created but not started.

RUNNABLE : ready or running.

BLOCKED : blocked on a lock.

WAITING : waiting without timeout.

TIMED_WAITING : waiting with timeout (e.g., Thread.sleep).

TERMINATED : execution finished.

A state transition diagram is shown below.

Problem Analysis

Thread‑pool exhaustion can stem from various cases:

Blocking issues (e.g., database connection failure) – look for thread --state showing BLOCKED or TIMED_WAITING.

Busy CPU‑bound work – look for RUNNABLE threads via thread -n.

GC pauses – require additional metrics beyond thread.

Targeted inspection – after grepping Dubbo threads, use thread ${thread_id} to view stack traces and identify the offending service.

Conclusion

The article uses Dubbo thread‑pool exhaustion as a case study to demonstrate how to analyze thread‑related problems and quickly diagnose them with Arthas, eliminating the need for manual jstack analysis.