Understanding Kafka Architecture: Topics, Partitions, Replication, Zero‑Copy, and Zookeeper Integration

Kafka's topic is a logical concept that can be divided into one or more partitions; each partition is a physical unit stored on different brokers. A topic can have multiple replicas, and each replica can be a leader or a follower. Producers send data to the leader partition, and followers replicate the data; consumers read from the leader.

Log data is appended to files called segments. Each segment consists of an .index file and a .log file. To avoid large log files and improve lookup efficiency, Kafka splits logs into segments and uses an index for fast access.

Kafka employs zero‑copy I/O to reduce data copies: the kernel transfers data directly from disk to the network socket without copying it through user‑space buffers, which improves performance and reduces context switches.

Zookeeper plays a crucial role in the Kafka cluster:

Broker registration: each broker creates a temporary node under /brokers/ids containing its IP and port.

Topic registration: topics are recorded under /brokers/topics, with each broker publishing the number of partitions it serves for a given topic.

Consumer registration: consumers create nodes under /consumers/[group_id]/ids/[consumer_id] and write their subscribed topics.

Consumer group coordination: watchers monitor changes in /consumers/[group_id]/ids to rebalance load when consumers join or leave.

Partition‑consumer mapping: the owner of a partition is stored under

/consumers/[group_id]/owners/[topic]/[broker_id-partition_id]

.

Offset tracking: each consumer records its progress in

/consumers/[group_id]/offsets/[topic]/[broker_id-partition_id]

.

Producer load balancing can be achieved either by simple four‑layer (TCP) load balancing, where a producer connects to a single broker, or by using Zookeeper to discover broker changes dynamically. Consumer load balancing follows a similar pattern, ensuring that each partition is consumed by only one consumer within a group while allowing a consumer to read from multiple partitions.

Overall, understanding these mechanisms—topic/partition design, replication, zero‑copy I/O, and Zookeeper coordination—helps developers diagnose issues, optimize performance, and design robust Kafka‑based streaming systems.

// Read file and send via socket (traditional I/O)
buffer = File.read
Socket.send(buffer)