Why Adding More Dubbo Connections Doesn’t Always Boost Throughput

Introduction

Dubbo’s connection‑control allows you to limit the number of long‑living TCP connections a consumer establishes with a provider. The default is a single connection, but the feature can be configured via the connections attribute.

Basic Demo

A simple demo with a consumer (192.168.4.226) and a provider (192.168.4.224) is set up using direct connections.

<dubbo:reference id="userService" check="false" interface="org.apache.dubbo.benchmark.service.UserService" url="dubbo://192.168.4.224:20880"/>

<dubbo:service interface="org.apache.dubbo.benchmark.service.UserService" ref="userService"/>
<bean id="userService" class="org.apache.dubbo.benchmark.service.UserServiceServerImpl"/>

After starting both sides, the TCP connections can be observed with: lsof -i:20880 (mac) netstat -ano | grep 20880 (linux)

Connection Control in Action

Setting connections="2" creates two long‑living connections instead of one:

<dubbo:reference id="userService" check="false" interface="org.apache.dubbo.benchmark.service.UserService" url="dubbo://192.168.4.224:20880" connections="2"/>

Observing the netstat output confirms the presence of two ESTABLISHED connections.

When to Use Multiple Connections

Benchmark tests were run with the official dubbo-benchmark project on two 4c8g ECS instances. The tests compared connections=1 and connections=2:

Benchmark           Mode  Cnt    Score    Error  Units
Client.createUser  thrpt   3  22265.286 ± 3060.319  ops/s
Client.existUser   thrpt   3  33129.331 ± 1488.404  ops/s
Client.getUser     thrpt   3  19916.133 ± 1745.249  ops/s
Client.listUser    thrpt   3   3523.905 ±  590.250  ops/s

Benchmark           Mode  Cnt    Score    Error  Units
Client.createUser  thrpt   3  31111.698 ± 3039.052  ops/s
Client.existUser   thrpt   3  42449.230 ± 2964.239  ops/s
Client.getUser     thrpt   3  30647.173 ± 2551.448  ops/s
Client.listUser    thrpt   3   6581.876 ±  469.831  ops/s

The initial results suggested a near‑doubling of throughput with two connections, but the methodology was flawed.

Root Cause of the Flawed Results

The benchmark changed two variables simultaneously: the number of connections and the number of Netty I/O worker threads. Dubbo’s Netty client uses Constants.DEFAULT_IO_THREADS (core + 1, capped at 32). On a 4c8g machine this defaults to 5 threads. When connections=1, only one I/O thread handles the channel; with connections=2, two I/O threads are active, giving an unfair advantage.

private static final EventLoopGroup NIO_EVENT_LOOP_GROUP = eventLoopGroup(Constants.DEFAULT_IO_THREADS, "NettyClientWorker");

int DEFAULT_IO_THREADS = Math.min(Runtime.getRuntime().availableProcessors() + 1, 32);

Similarly, the server’s Netty worker group defaults to the same value unless overridden via the iothreads attribute in the <dubbo:protocol> configuration.

<dubbo:protocol name="dubbo" host="${server.host}" server="netty4" port="${server.port}" iothreads="5"/>

Corrected Benchmark

To isolate the effect of connection count, the I/O thread count was fixed to 1 on both client and server (the client required a custom build to expose the setting). The revised results:

Benchmark           Mode  Cnt    Score    Error  Units
Client.createUser  thrpt   3  22265.286 ± 3060.319  ops/s
Client.existUser   thrpt   3  33129.331 ± 1488.404  ops/s
Client.getUser     thrpt   3  19916.133 ± 1745.249  ops/s
Client.listUser    thrpt   3   3523.905 ±  590.250  ops/s

Benchmark           Mode  Cnt    Score    Error  Units
Client.createUser  thrpt   3  21776.436 ± 1888.845  ops/s
Client.existUser   thrpt   3  31826.320 ± 1350.434  ops/s
Client.getUser     thrpt   3  19354.470 ±  369.486  ops/s
Client.listUser    thrpt   3   3506.714 ±   18.924  ops/s

With identical I/O threads, increasing connections from 1 to 2 produced no meaningful throughput gain, confirming that connection count alone is not a performance lever.

Conclusion

Configuration parameters are not universally beneficial; blindly increasing them can mislead. In most production environments the number of connections far exceeds the number of I/O threads, so the default single‑connection setting is usually sufficient. Only when a specific use‑case demands more connections should the feature be tuned, and even then it must be validated with controlled benchmarks.