Unveiling Dubbo’s IO Journey: From Consumer Call to Provider Response

Hello, everyone! I’m Xiao Lou, and today I’m sharing an entertaining yet technical walkthrough of Dubbo’s IO process originally written by a colleague.

Some Interesting Questions

Does the data packet leave immediately after the business method executes?

What are the differences between Netty 3 and Netty 4 thread models?

What happens to the packet once it reaches the OS socket buffer?

Why does the Provider log show low latency while the Consumer times out, and how can we debug it?

Is the packet simply sent down a physical pipe?

When does the Consumer’s business thread awaiting on a Condition get awakened?

…

We will use Dubbo 2.5.3 as the Consumer and Dubbo 2.7.3 as the Provider, narrating the whole interaction from the packet’s perspective.

Interesting Journey

1. Dubbo 2.5.3 Consumer Initiates Request

I am a data packet born in the Dubbo 2.5.3 Consumer town, tasked with delivering information.

The business thread invokes a method, which triggers FailoverClusterInvoker#doInvoke to select a Provider, passes through various Consumer Filters, then traverses Netty 3’s pipeline and finally reaches NioWorker#scheduleWriteIfNecessary, entering the writeTaskQueue.

Meanwhile the main thread waits on a Condition inside DefaultFuture.

The Netty 3 IO worker thread continuously runs its loop, processing the write task.

Eventually NioWorker#processWriteTaskQueue picks me up, writes me to the OS socket buffer, and I wait there while two “travel groups” – the main thread and the Netty 3 IO worker – handle me efficiently.

2. Operating System Sends the Packet

Inside the OS socket buffer, the packet undergoes several transformations:

Transport layer adds source and destination ports.

Network layer adds source and destination IPs and determines the next‑hop IP via subnet masking.

Data‑link layer uses ARP to add the next‑hop MAC address and source MAC address.

Each “cable car” hop updates MAC addresses; large packets may be split (fragmented) and later reassembled (sticky packets). Congestion can cause the packet to wait.

3. Experience on the Provider Side

Arriving at the Provider, I board a “zero‑copy” boat and enter Netty 4, passing through NioEventLoop#processSelectedKeys and various inbound handlers until reaching the AllChannelHandler thread pool.

There I am decoded by NettyCodecAdapter#InternalDecoder, which also handles fragmentation and reassembly.

4. Provider Generates a New Packet

After the Provider’s business method finishes, the new packet is written via io.netty.channel.AbstractChannelHandlerContext#writeAndFlush, scheduled by io.netty.util.concurrent.SingleThreadEventExecutor#execute, placed into the task queue, and awaits dispatch by the NioEventLoop.

5. Arrival at Dubbo 2.5.3 Consumer

The packet reaches the Consumer’s OS socket buffer, hops onto another “zero‑copy” boat, and finally arrives at the Consumer’s business thread pool after being read by NioWorker#run, NioWorker#processSelectedKeys, and NioWorker#read.

At this point DefaultFuture#doReceived is executed, the waiting Condition is signaled, and the blocked thread on the Consumer side is awakened, completing the packet’s mission.

Summary of Netty 3 vs Netty 4 Thread Models

Both Netty 3 and Netty 4 share the same read process, where the pipeline is executed by the I/O thread. The key difference lies in the write process: Netty 3 executes the pipeline on the business thread, while Netty 4 executes it uniformly on the I/O thread, reducing context‑switch overhead but limiting concurrent handler execution.

Typical Troubleshooting Scenarios

When Provider logs show normal latency but the Consumer times out, consider the following checks:

Monitor the outermost deserialization method org.apache.dubbo.remoting.transport.DecodeHandler#received to rule out business‑method latency.

Inspect whether packet writing on the Provider side is slow by monitoring io.netty.channel.AbstractChannelHandlerContext#invokeWrite.

Use netstat to examine TCP socket queues (Recv‑Q and Send‑Q) for potential backlog.

Check Consumer’s NioWorker#processSelectedKeys (Dubbo 2.5.3) for high latency.

Trace the entire link to pinpoint bottlenecks.

Note: Netty 3’s read process is identical to Netty 4’s; the pipeline is always executed by the I/O thread.