How Kafka Broker Handles High‑Throughput Requests: Deep Dive into Its Network Architecture

01 Overview

Understanding Kafka Broker's request handling starts with the basic Request/Response model used by producers, consumers, and brokers.

02 Sequential Processing Model

The simplest implementation uses a while loop that continuously accepts producer requests and stores them to disk, but it suffers from request blocking and poor throughput.

Request blocking: Each request must wait for the previous one to finish. Poor throughput: Lack of concurrency makes it unsuitable for high‑frequency traffic.

03 Multi‑Threaded Asynchronous Model

Introducing a dedicated thread per request (connection‑per‑thread) improves throughput and eliminates blocking, yet the overhead of creating a thread per request becomes a bottleneck under high concurrency.

Higher throughput: Improves server capacity compared to the sequential model. Non‑blocking: Requests are processed asynchronously.

However, the thread‑per‑request approach still cannot meet the demands of ultra‑high‑throughput systems.

04 Reactor Pattern

The Reactor pattern, based on event‑driven design, uses a dispatcher (Acceptor) to accept connections and multiple request handlers to process them. It leverages Java NIO's Selector for multiplexing.

Key components include:

Acceptor thread: Accepts TCP connections and registers OP_ACCEPT events.

Processor threads: Handle OP_READ/OP_WRITE events, manage new connections, and process responses.

Multiple Selectors can be introduced to avoid a single selector becoming a bottleneck.

05 Kafka High‑Concurrency Network Architecture

The architecture consists of two main components: SocketServer and RequestHandlerPool . SocketServer contains Acceptor, Processor, and RequestChannel, implementing the Reactor pattern to handle client requests.

Acceptor creates a NIO Selector, opens a ServerSocketChannel, and registers OP_ACCEPT. It also spawns a pool of Processor threads based on num.network.threads.

Processor threads manage three queues: newConnections, inflightResponse, and responseQueue, handling connection setup, response buffering, and final response delivery.

05.1 Acceptor Thread

/**
 * Thread that accepts and configures new connections. There is one of these per endpoint.
 */
private class Acceptor(val endPoint: EndPoint,
                      val sendBufferSize: Int,
                      val recvBufferSize: Int,
                      brokerId: Int,
                      connectionQuotas: ConnectionQuotas,
                      metricPrefix: String) extends AbstractServerThread(connectionQuotas) with KafkaMetricsGroup {
  private val nioSelector = NSelector.open()
  val serverChannel = openServerSocket(endPoint.host, endPoint.port)
  private val processors = new ArrayBuffer[Processor]()
  def run(): Unit = {
    serverChannel.register(nioSelector, SelectionKey.OP_ACCEPT)
    startupComplete()
    var currentProcessorIndex = 0
    while (isRunning) {
      val ready = nioSelector.select(500)
      if (ready > 0) {
        accept(key).foreach { socketChannel =>
          val processor = synchronized { processors(currentProcessorIndex) }
          currentProcessorIndex += 1
        }
      }
    }
  }
}

The Acceptor uses a Selector to monitor OP_ACCEPT events, creates Processor threads, and distributes incoming connections among them.

05.2 Processor Thread

override def run(): Unit = {
  startupComplete()
  while (isRunning) {
    configureNewConnections()
    processNewResponses()
    poll()
    processCompletedReceives()
  }
}

val ConnectionQueueSize = 20
private val newConnections = new ArrayBlockingQueue[SocketChannel](ConnectionQueueSize)
private val inflightResponses = mutable.Map[String, RequestChannel.Response]()
private val responseQueue = new LinkedBlockingDeque[RequestChannel.Response]()

Processor threads handle new connections, send responses, poll for I/O events, and manage request/response queues.

06 Core Request Processing Flow

Clients send requests to the Acceptor thread.

Acceptor creates a NIO Selector, registers OP_ACCEPT, and opens ServerSocketChannel.

Acceptor spawns Processor threads (configured by num.network.threads) and enqueues connections.

Processor threads poll for ready I/O events (OP_READ/OP_WRITE).

Received data is assembled into Request objects and placed in the RequestChannel's RequestQueue.

KafkaRequestHandler threads (configured by num.io.threads) dequeue Requests and invoke KafkaApis.handle to process them.

After processing, responses are enqueued to the Processor's response queue and sent back to clients.

07 System Tuning

Key broker parameters for performance:

num.network.threads: Set to CPU cores × 2 to avoid network idle.

num.io.threads: Set to disk count × 2 for optimal request handling.

num.replica.fetchers: Set to CPU cores / 4 to improve follower‑leader sync.

compression.type: Use lz4 for better CPU utilization and reduced network traffic.

queued.max.requests: Configure at least 500 for production workloads.

log.flush.* parameters: Keep defaults; let the OS manage flushing.

auto.leader.rebalance.enable: Disable to prevent unpredictable load spikes.

08 Summary

The article covered Kafka's high‑performance server design principles, evolving from simple sequential handling to the Reactor pattern, detailed the roles of Acceptor and Processor threads, explained the full request processing pipeline, and provided practical broker‑side tuning recommendations.