Understanding Kafka's Core Design: Topics, Partitions, Consumer Groups, and Cluster Architecture

Kafka is presented as a high‑performance distributed messaging system that acts like a warehouse, providing buffering and decoupling between producers and consumers.

1. Kafka Basics

Messages are stored on disk rather than purely in memory, and the system uses topics (analogous to database tables) to categorize streams of data.

Each topic is divided into multiple partitions , which are physical directories on broker machines; partitions improve throughput by allowing parallel processing.

Key components include:

Producer : sends messages to a topic.

Consumer : reads messages from a topic.

Message : the unit of data processed by Kafka.

2. Kafka Cluster Architecture

A topic can have several partitions spread across different brokers. Replication is used to avoid data loss; each partition can have multiple replicas, with one acting as the leader and the others as followers .

Consumers belong to a Consumer Group . Only one consumer in a group can read a particular partition, ensuring no duplicate consumption. conf.setProperty("group.id", "tellYourDream") Different consumer groups can read the same topic independently:

consumerA:<br/>group.id = a<br/>consumerB:<br/>group.id = a<br/><br/>consumerC:<br/>group.id = b<br/>consumerD:<br/>group.id = b

The controller node, elected via Zookeeper, manages cluster metadata, broker registration, and partition assignments.

Performance Highlights

Sequential Write : Kafka appends data to the end of log files, achieving near‑memory speeds on spinning disks.

Zero‑Copy : Uses Linux sendFile to transfer data directly from disk to network sockets, eliminating extra memory copies.

Log Segmentation : Each partition’s .log file is limited to 1 GB; when full, a new segment is created (log rolling), improving read/write efficiency.

00000000000005367851.index<br/>00000000000005367851.log<br/>00000000000005367851.timeindex

Network Design : Requests first hit an Acceptor, then are dispatched round‑robin to a pool of processor threads, which hand them off to a thread pool for actual I/O processing, forming a three‑layer reactor model.