Mastering Kafka: Deep Dive into Architecture, Production, Consumption, and Transactions

Overview

Kafka is a distributed message queue offering high performance, persistence, replication, and horizontal scalability. Producers write messages to topics, consumers read them, enabling decoupling, peak‑shaving, and asynchronous processing.

Production

Producers create records specifying a topic, value, optional key, and optional partition. Records are serialized and batched before being sent; a send does not immediately trigger a network packet.

If the partition is omitted, Kafka assigns it based on the key:

Key present: hash the key; the same key maps to the same partition (subject to change when partitions increase).

No key: round‑robin selection.

Requests targeting the same partition are aggregated and dispatched by a dedicated thread.

API

Kafka offers a High‑Level API that abstracts offset handling and routing, and a Simple API where the application must manage offsets manually.

Partition Management

Partitions are distributed across brokers, with each partition having a leader that handles all client requests and replicates data to followers. When a broker fails, leaders on that broker are re‑elected.

The controller, elected via ZooKeeper, is responsible for partition assignment and leader election.

Leader Election

Sort all brokers and partitions.

Assign partition i to broker (i mod n) as leader.

Assign replica j of partition i to broker ((i + j) mod n).

Consumption

Consumers join a consumer group; each group can have multiple consumers, but a single partition is consumed by only one consumer within the group. Different groups may consume the same partition concurrently.

Offsets were originally stored in ZooKeeper, but since Kafka 0.10 they are stored in an internal __consumer_offsets topic with compacted cleanup, using a key composed of groupId, topic, and partition.

The partition for a given consumer group is calculated as:

__consumer_offsets partition = Math.abs(groupId.hashCode() % groupMetadataTopicPartitionCount)

where groupMetadataTopicPartitionCount defaults to 50.

Rebalance Process

Consumer sends ConsumerMetadataRequest to any broker to discover its coordinator.

Coordinator assigns partitions; consumers send JoinGroupRequest and one consumer becomes leader.

Leader distributes partition assignments via SyncGroupRequest.

Coordinator notifies all consumers of the final assignment.

Rebalance occurs when partitions or consumer counts change, or when consumers or the coordinator fail.

Message Delivery Semantics

At most once: messages may be lost but never duplicated.

At least once: messages are never lost but may be duplicated.

Exactly once: no loss and no duplication, supported from Kafka 0.11 when downstream is also Kafka.

Most applications use “at least once” and implement idempotency themselves.

Producer Idempotence

Each producer is assigned a unique PID and a monotonically increasing sequence number. The broker only accepts a message if req_seq == broker_seq + 1, ensuring ordering and preventing duplicates.

Transactional Messaging

Transactions involve a transaction ID (Tid) supplied by the application and a Transaction Coordinator that records transaction state in a compacted log.

Typical flow:

Client requests a broker to locate the Transaction Coordinator.

Coordinator returns the PID and epoch for the Tid.

Client writes data to target partitions and records offset updates.

Client sends a Commit request; the coordinator logs PrepareCommit or PrepareAbort and sends marker messages to partitions.

After all markers are written, the transaction is considered committed.

During the prepare phase, uncommitted messages are invisible to consumers; Kafka filters them client‑side to preserve zero‑copy performance.

File Organization

Data is stored as files on the filesystem. Each topic contains partitions; each partition contains segments (log files) named by the smallest offset in the segment, with corresponding .index files for offset and time lookup.

Indexes use sparse matrices to reduce size, storing a base offset and the byte position within the segment.

Common Configuration

Key broker and topic settings include replication factor, min.insync.replicas, log segment size, retention policies (size‑based and time‑based), and cleanup policies (delete vs. compact). Since Kafka 0.10, log cleanup uses the timestamp of the latest message rather than file modification time.