Kafka Fundamentals: Topics, Partitions, Producers, Consumers, and Cluster Architecture

Kafka organizes messages into topics, each of which can be split into multiple partitions to enable parallel processing, improve scalability, and increase throughput. Within a partition, messages are strictly ordered and identified by an offset.

A partition maps to a file on disk; incoming data is appended directly without intermediate buffering.

Unlike many messaging systems that delete messages after consumption, Kafka retains data based on configurable time policies, allowing messages to be reprocessed later.

Producers decide which partition to write to, using strategies such as round‑robin or key‑based hashing; if the key is null, round‑robin is applied.

Consumers maintain their own offset and belong to consumer groups. Within a group, each consumer reads from distinct partitions, ensuring that each message is processed only once per group (queue model). Different consumer groups operate independently, providing both broadcast (multiple groups) and unicast (single group) consumption patterns.

A Kafka cluster consists of multiple brokers, each managing several partitions. One broker is elected as the Controller, which handles broker registration, partition replica assignment, and leader election, all coordinated via Zookeeper.

Topics act like databases, partitions like tables, and brokers like servers. Topics can have multiple consumer groups, and each partition has one leader and one or more followers. The leader handles all read/write requests, while followers replicate data for fault tolerance.

Replication ensures high availability: each partition can have multiple replicas; if the leader fails, a follower is promoted to leader. This mechanism, combined with Zookeeper‑based metadata, provides robust cluster management.

Offsets uniquely identify messages within a partition and are stored either in Zookeeper (pre‑0.9) or in the internal "__consumer_offsets" topic (0.9 and later). Consumers control their offset progression, allowing them to replay or skip data as needed.

Kafka retains all published records for the configured retention period, making it suitable for long‑term storage and large‑scale data pipelines.