Understanding How Kafka Producer Sends Messages to the Broker: Components, Caching Model, Partitioning, and Sender Thread

The article introduces the Kafka producer workflow, outlining four key aspects: client components, client cache storage model, partition determination, and the sender thread's operation.

Client components include KafkaProducer (the producer process), RecordAccumulator (collects messages into a local buffer), Sender (a dedicated thread that prepares batches for transmission), and Selector (handles network I/O).

Messages are first serialized into binary form and stored in a double‑ended queue managed by the RecordAccumulator. The cache stores messages in batches; each batch holds up to N records, and new records are appended on the left side of the queue.

When determining the target partition , messages with a key use the hash function Utils.murmur2(key) % numPartitions , while key‑less messages are assigned using a round‑robin strategy. Only partitions that currently have a leader are considered.

The Sender thread scans the accumulator for batches that have reached the configured size, maps partition → batch to BrokerId → batches , creates a client request for each broker, and hands it to the NetworkClient . The NetworkClient then performs three main actions: ready() (establish connections), send() (queue requests on channels), and poll() (actually transmit requests and receive responses).

A comparison of partition‑based versus broker‑based network models shows that aggregating batches per broker reduces the number of network connections, which Kafka prefers.

In summary, the producer’s journey from message creation to broker delivery involves serialization, buffering, partition calculation, batch preparation by the Sender, and network transmission by the NetworkClient, providing a clear view of Kafka’s internal messaging pipeline.