Mastering Non‑Blocking IO Pipelines in Java: Design Challenges and Solutions

Non‑Blocking IO Pipeline Overview

A non‑blocking IO pipeline consists of a chain of components that use Selector to detect when a Channel has readable data, read the data, transform it, and write the output back to the channel. The pipeline can have multiple components and may read from several channels simultaneously.

Blocking vs. Non‑Blocking Pipelines

Blocking pipelines read from streams (e.g., InputStream) one byte at a time, blocking until data is available. This simplifies message readers and writers but requires a dedicated thread per connection, which does not scale to millions of concurrent connections due to memory consumption (e.g., 1 TB for 1 000 000 threads).

To reduce thread count, servers often use a thread pool that pulls connections from a queue, but idle or slow connections can still block the pool, leading to latency or unresponsiveness.

Basic Non‑Blocking Design

Using a single thread with a Selector, the server registers multiple SelectableChannel instances. When select() or selectNow() returns, only channels with readable data are processed, eliminating unnecessary polling.

Reading Partial Messages

Data read from a channel may contain less than one full message, exactly one, or multiple messages. The challenges are detecting complete messages and buffering incomplete ones until the remaining bytes arrive.

Detect whether a data block contains a complete message.

Store partial messages until the rest arrives.

Two main buffer strategies are discussed:

Per‑Reader Fixed Buffer

Allocate a buffer large enough for the maximum message size (e.g., 1 MB) per reader. This approach is simple but wasteful at scale.

Resizable Buffer

Start with a small buffer (e.g., 4 KB) and grow it only when a larger message arrives, reducing memory usage. Three size tiers (4 KB, 128 KB, max) can limit copying overhead while keeping most messages in the smallest buffer.

Append‑Resize Buffer

Maintain a list of byte arrays or slices, appending new arrays as needed. This avoids copying but stores data in non‑contiguous memory, complicating parsing.

TLV‑Encoded Messages

Some protocols use TLV (type‑length‑value) framing, allowing immediate allocation of the exact message size. However, large messages from slow connections can still exhaust memory, so field‑level allocation or timeout strategies are recommended.

Writing Partial Messages

A MessageWriter tracks how many bytes of each message have been written. If the writer cannot send all queued messages, it buffers them and registers the channel with the selector only when there is data to write.

The two‑step write management is:

When a message is queued, register its channel with the selector if not already registered.

When the server has capacity, poll the selector for writable channels and let each associated writer flush its data; deregister the channel once all data is sent.

This ensures only channels that actually need to write are monitored, reducing selector overhead.

Server Loop Composition

The non‑blocking server repeatedly executes three pipelines:

Read pipeline – checks for new inbound data on open connections.

Process pipeline – handles any complete messages that have been assembled.

Write pipeline – attempts to flush pending outbound messages.

Optimizations can skip pipelines when there is no work to do.

Thread Model

The reference implementation uses a two‑thread model: one thread accepts inbound connections via ServerSocketChannel, and a second thread runs the read‑process‑write loop for all connections.

For a complete example, see the accompanying GitHub repository, which contains the source code and a runnable demo.