Kafka Overview: Architecture, Storage Mechanism, Replication, and Consumer/Producer Model

Kafka, initially created by LinkedIn, is a distributed, partitioned, multi‑replica message system that relies on Zookeeper for coordination. It can handle massive real‑time data streams for use cases such as log collection, event sourcing, user activity tracking, and offline data loading into Hadoop or data warehouses.

The platform provides high throughput (hundreds of thousands of messages per second) and low latency (few milliseconds) by using sequential disk writes (append‑only logs), batch processing, and optional compression (gzip, snappy). Each topic is divided into partitions, each stored as a directory of segment files; a partition’s offset uniquely identifies each message.

Producers send messages to the leader of a partition; the leader writes to its local log and replicates to follower replicas. Delivery guarantees are configurable via the acks setting: acks=0 (fire‑and‑forget), acks=1 (leader acknowledgment), and acks=-1 (all in‑sync replicas acknowledgment), supporting at‑most‑once, at‑least‑once, and exactly‑once semantics.

Consumer groups enable horizontal scaling: each partition is consumed by only one consumer within a group, while multiple groups can read the same topic independently. Offsets are tracked either in Zookeeper (high‑level API) or locally by the consumer (low‑level API), allowing replay or manual offset management.

Replication is managed by assigning leaders and followers across brokers using a deterministic algorithm: partition i is placed on broker (i mod n), and its j‑th replica on broker ((i + j) mod n). The in‑sync replica (ISR) set ensures that a message is considered committed only after being written to all ISR members, providing fault tolerance even when some brokers fail.

Leader election occurs when a broker or its controller fails; the controller selects a new leader from the ISR set, favoring brokers with fewer leader partitions to balance load. If all replicas become unavailable, Kafka can either wait for an ISR member to recover or promote the first recovered replica, trading availability for consistency.

Zookeeper stores metadata such as broker registrations, topic‑partition mappings, consumer group membership, and offset information. Watches on these znodes trigger rebalancing when brokers or consumers join or leave, ensuring balanced partition assignment.

Operational best practices include matching the number of partitions to the number of consumer threads for optimal throughput, configuring appropriate replication factors (typically ≥2), monitoring ISR health, and tuning segment size and retention policies to balance storage usage and performance.