Understanding Kafka Architecture: Topics, Partitions, Replication, Consumers, Network Design, Zero‑Copy and Zookeeper

Grasping the underlying principles of Kafka is crucial for effective troubleshooting; interview questions focus on concepts rather than commands, and a solid understanding enables rapid issue localization.

Topic : A logical entity; the example creates TopicA with three partitions distributed across different brokers.

Partition & Replication : Each partition is a physical unit; setting a replication factor of 3 creates three identical replicas for each partition, stored on separate brokers.

Log Segmentation : To avoid oversized log files, Kafka splits each partition into multiple segments, each consisting of an .index file and a .log data file.

Leader & Follower : Among replicas, one is elected leader; producers write to the leader, and followers replicate from it. Consumers also read from the leader.

Consumer & Consumer Group : A consumer group consists of one or more consumer instances; each partition is consumed by only one member of the group, while a single consumer can read from multiple partitions.

Kafka Network Design : Client requests go to an Acceptor, which hands them off to a pool of processor threads (default 3) in a round‑robin fashion. ReaderThreadPool (default 8 threads) processes requests, generates responses, and processors return data to clients, forming an enhanced reactor model.

Zero‑Copy I/O : Traditional I/O involves four data copies (disk → kernel buffer → application buffer → socket buffer → NIC). Kafka uses zero‑copy so the kernel transfers data directly from disk to the socket, reducing CPU overhead and context switches.

//读取文件，再用socket发送出去
buffer = File.read

Socket.send(buffer)

Zookeeper in a Kafka Cluster : Zookeeper registers brokers under /brokers/ids, tracks topic‑to‑broker mappings under /brokers/topics, manages consumer group membership under /consumers/[group_id]/ids, records partition‑consumer ownership via

/consumers/[group_id]/owners/[topic]/[broker_id-partition_id]

, and stores consumer offsets at

/consumers/[group_id]/offsets/[topic]/[broker_id-partition_id]

. It also facilitates dynamic load balancing for producers and consumers.