Why Dubbo Fails Under C10K Load and How to Fix It

Background and Motivation

Dubbo is a lightweight open‑source Java RPC framework widely used for building distributed services. China Construction Bank began migrating to a distributed architecture in 2014 and built a Dubbo‑based platform. As banking services become increasingly online, diverse, and intelligent, the platform must support thousands of providers and tens of thousands of consumers, raising the classic C10K challenge.

Test Environment Setup

Using Dubbo 2.5.9 (Netty 3.2.5.Final), a provider service with a 100 ms sleep and a 5 s timeout was deployed. One 8‑CPU/16 GB server hosted the provider container, while hundreds of similar servers hosted 7,000 consumer containers, each invoking the service once per minute. The Dubbo monitoring center tracked call metrics.

Validation Scenarios

Two scenarios were executed:

Scenario 1: Start the provider, then launch consumers in batches. Observe transaction outcomes for one hour.

Scenario 2: After stable operation, restart the provider and watch its behavior.

Both scenarios exhibited frequent transaction timeouts, confirming that Dubbo struggled under C10K conditions.

Root‑Cause Analysis – Scenario 1

Network captures showed that after a consumer request reached the provider, the provider took over 2 seconds to begin processing, and another 2 seconds to send the response, exceeding the 5 s timeout. GC logs and jstack traces revealed no abnormal pauses, ruling out GC or thread deadlock.

Further investigation identified that Netty worker threads were constantly handling heartbeat packets (sent every 60 s) from thousands of consumers, causing CPU spikes and thread saturation. This heartbeat‑induced load delayed business request handling, leading to timeouts.

Root‑Cause Analysis – Scenario 2

When the provider restarted, it immediately responded to incoming consumer requests with RST packets, causing massive timeouts. The underlying issue was a full TCP SYN backlog (default 50 in Dubbo, 128 in Linux) that overflowed when thousands of consumers attempted to reconnect simultaneously, creating one‑sided connections.

Observations showed that the backlog overflow prevented the provider from accepting new connections promptly, and the kernel’s SYN retransmission logic caused additional delays before connections could be established.

Optimization Measures

The team implemented a series of system‑level and Dubbo‑level tweaks:

Increase the TCP full‑connection queue size (e.g., raise net.core.somaxconn and Dubbo’s backlog).

Adjust Netty’s epoll model to improve accept speed.

Bypass serialization for heartbeat messages.

Increase Netty I/O thread count (from 9 to 20).

Distribute heartbeat traffic to avoid bursts.

These changes yielded measurable improvements: CPU peaks dropped 20‑30 %, P99 latency fell from 191 ms to 125 ms, and after provider restart no transaction timeouts were observed even with 7,000 consumers.

Results and Production Impact

After applying the optimizations, the bank’s production environment now runs scenarios with tens of thousands of consumers per provider without timeout incidents. The enhancements have been contributed back to the Dubbo community.

Future Outlook

Continued collaboration with the Dubbo community aims to further boost performance, reliability, and scalability for financial‑grade deployments.