Why Dubbo Times Out Under C10K Load and How to Fix It

Background

Dubbo is a lightweight open‑source Java RPC framework widely adopted by enterprises for building distributed service architectures. Since 2014, Industrial and Commercial Bank of China (ICBC) has built a distributed service platform based on Dubbo.

Test Setup

Using Dubbo 2.5.9 (Netty 3.2.5.Final), a provider service with a 100 ms sleep and a consumer timeout of 5 s were deployed. One 8‑core/16 GB server ran the provider in containers, while hundreds of similar servers ran 7 000 consumer instances. Dubbo’s monitoring center was enabled to track calls.

Validation Scenarios

Scenario 1 : Start the provider first, then launch consumers in batches. After one hour, occasional transaction timeouts were observed, with consumers spread across many servers.

Scenario 2 : After stable operation, restart the provider. Within 1‑2 minutes after restart, a large number of transaction timeouts occurred.

Both scenarios demonstrated that Dubbo could not reliably handle the C10K load.

Problem Analysis

Initial suspicion focused on process GC pauses or network latency, but GC logs, jstack traces, and packet captures showed no anomalies. The real issues were identified as:

Provider‑side processing delays exceeding 2 seconds before handling requests, causing total call time to surpass the 5 s timeout.

Excessive heartbeat traffic (every 60 s) saturating Netty worker threads, leading to CPU spikes and delayed business packet processing.

TCP full‑connection queue overflow (default backlog 50) during provider restart, creating half‑open connections and RST packets that trigger client‑side timeouts.

Scenario 1: Provider‑Side Delay

Network captures showed that after the provider received a request, it took over 2 seconds before invoking the business method, and another 2 seconds before sending the response, resulting in timeout.

Further investigation revealed:

Netty worker threads handling heartbeats every ~60 seconds, keeping all 9 workers (on an 8‑core box) busy.

CPU usage spikes correlated with heartbeat bursts.

Network receive/send queues accumulated packets around the long‑delay moments.

These observations confirmed that dense heartbeat traffic was monopolizing Netty workers and inflating transaction latency.

Scenario 2: Half‑Open Connections

When the provider restarted, the TCP three‑way handshake often failed because the full‑connection queue (size 50) was full, leading to half‑open connections. The provider responded with RST packets, and consumers threw “Connection reset by peer” exceptions, causing timeouts.

Analysis of the TCP backlog showed that the kernel parameter net.core.somaxconn (default 128) and Dubbo’s default backlog (50) limited the queue, and the massive simultaneous reconnection attempts after a restart overflowed it.

Root Causes

1. Heartbeat mechanism causing Netty worker thread saturation. 2. Insufficient TCP full‑connection queue capacity leading to half‑open connections during provider restarts.

Proposed Improvements

To mitigate the issues, the following ideas were explored:

Reduce per‑heartbeat processing time.

Increase the number of Netty worker threads.

Scatter heartbeat sending to avoid bursts.

Expand the TCP backlog and improve accept speed.

Optimization Measures and Effects

The team applied a series of optimizations at both the system and Dubbo framework levels:

TCP backlog expansion : eliminated post‑restart timeouts.

Epoll model tuning : reduced backlog overflow and improved accept speed.

Heartbeat serialization bypass : removed CPU spikes, lowered peak CPU by 20%, reduced average request‑response gap from 27 ms to 3 ms, and cut P99 latency from 191 ms to 133 ms.

Increase iothreads (from 9 to 20): further reduced average gap to 14 ms and P99 latency to 186 ms.

Heartbeat scattering on provider and consumer : decreased heartbeat packet peaks from ~15 k/s to a few thousand per second.

Comprehensive Validation

In the 1‑provider‑to‑7 000‑consumer test, after applying all optimizations, the provider’s CPU peak dropped by 30%, the average processing gap stayed within 1 ms, and P99 latency fell to 125 ms. No transaction timeouts were observed during long‑run tests.

Production Results

ICBC integrated these enhancements into its production distributed service platform. Even with a provider handling tens of thousands of consumers, the system runs without timeout incidents, meeting financial‑grade performance requirements.

Future Outlook

ICBC will continue contributing the improvements back to the Dubbo community, aiming to further boost performance, high‑availability, and support large‑scale financial digital transformation.