Deep Dive into Kafka Architecture: Topics, Partitions, Replication, Consumers, and Transactions

Kafka is a distributed message queue offering high performance, persistence, replication, and horizontal scalability; producers write messages to topics while consumers read them, enabling decoupling, throttling, and asynchronous processing in system design.

Each topic consists of multiple partitions for horizontal scaling; adding partitions expands capacity, and messages within a partition remain ordered.

Producers create records specifying topic, value, optional key and partition. Records are serialized, batched, and sent to the appropriate broker; if no partition is provided, keys are hashed to a partition or round‑robin is used.

Kafka provides a High‑Level API that handles offsets, routing, etc., and a Simple API where the application must manage offsets manually.

Partitions are replicated across brokers; one replica acts as the leader handling all reads/writes, while followers sync from the leader. A controller elected via ZooKeeper manages partition allocation and leader election.

When a broker fails, the controller re‑elects leaders for affected partitions using the ISR (in‑sync replica) list stored in ZooKeeper.

Consumer groups subscribe to topics; each group can have multiple consumers, but a single partition is consumed by only one consumer within the group. If consumers exceed partitions, some remain idle.

Offsets were originally stored in ZooKeeper, but since version 0.10 they are persisted in an internal __consumer_offsets topic using compacted logs. The partition for a group's offsets is calculated as:

__consumers_offsets partition = Math.abs(groupId.hashCode() % groupMetadataTopicPartitionCount) // groupMetadataTopicPartitionCount defaults to 50

Rebalancing occurs when partitions or consumers change, involving coordinator selection, JoinGroup, SyncGroup, and partition assignment.

Kafka supports three delivery semantics: at‑most‑once (possible loss, no duplication), at‑least‑once (no loss, possible duplication), and exactly‑once (no loss, no duplication, available from 0.11 when downstream is also Kafka).

Exactly‑once is achieved by combining producer id (PID) and transaction id (TID) with a Transaction Coordinator that logs transaction states. The flow includes Begin, Prepare Commit/Abort, marker messages, and final Commit/Abort.

File organization stores logs as segments under each partition directory, with accompanying offset and time index files. Sparse indexing reduces memory usage, and binary search locates offsets within segments.

Key configuration areas include broker settings (e.g., replication factor, log retention) and topic settings (e.g., partitions, cleanup policy). Log cleanup respects size and time thresholds, ensuring active segments are retained.