Inside Kafka’s Network Stack: How SocketServer, Acceptor, and Processor Work

Kafka Network Communication Overview

The Kafka broker’s network layer is built around a SocketServer component that coordinates incoming client connections, request handling, and response delivery using a classic Reactor pattern.

SocketServer

AbstractServerThread : Base class for Acceptor and Processor threads.

Acceptor Thread : Listens for external TCP connections, creates a single Acceptor per SocketServer, and forwards received requests to downstream Processor threads via RequestChannel.

Processor Thread : Default number defined by num.network.threads; each Processor pulls requests from the channel, processes them, and returns responses.

Supporting components include KafkaRequestHandlerPool (I/O thread pool), ConnectionQuotas , TooManyConnectionsException , and the SocketServer class itself, which manages the lifecycle of Acceptor and Processor threads.

Acceptor Thread Details

The Acceptor implements the Dispatcher role of the Reactor pattern: it uses a Java NIO Selector to monitor SelectionKey.OP_ACCEPT events, accepts new TCP connections, and assigns each to a Processor thread.

Key Parameters

endPoint

: Broker connection string, e.g., PLAINTEXT://localhost:9092.

sendBufferSize

and recvBufferSize: Control the size of the socket send/receive buffers; increase them (default 100KB) for high‑latency environments (RTT > 10 ms).

If clients experience large network latency (RTT > 10 ms), increase sendBufferSize and recvBufferSize as the default 100 KB is often too small.

Acceptor Custom Attributes

nioSelector : Java NIO Selector instance used for event multiplexing.

processors : Thread pool of Processor threads created during Acceptor initialization.

Processor Thread API

addProcessors / removeProcessors : Dynamically manage the Processor thread pool.

run method: Executes the Reactor dispatch loop for handling assigned requests.

Processor Thread Internals

newConnections : Queue of newly accepted SocketChannel objects awaiting configuration.

inflightResponses : Temporary queue for responses that need post‑send callbacks.

responseQueue : Per‑Processor queue of responses to be returned to requesters.

Processing Workflow

configureNewConnections

Handles newly accepted connections; each Processor maintains its own Selector to register and configure the SocketChannel.

processNewResponses

Sends responses back to the originating client and places them into the temporary inflight queue.

poll

Invokes the NIO poll operation to process ready I/O events for the Processor.

processCompletedReceives

Receives and processes incoming Request objects.

processCompletedSends

Handles completion of send operations.

processDisconnected

Manages disconnection events and cleans up associated resources.

closeExcessConnections

Closes connections that exceed configured limits to protect the broker.