Understanding Kafka: Core Design, Architecture, and Performance

Understanding Kafka: Core Design

We discuss the excellent design of Kafka to improve design and coding skills.

1. Kafka Basics

Message systems act like a warehouse, providing caching and decoupling between producers and consumers.

Example scenario: telecom operators (China Mobile, China Unicom, China Telecom) outsource log processing; Kafka can be used to collect logs for user‑profile analysis.

Kafka stores data on disk but behaves like a cache; a topic is analogous to a relational‑database table.

To retrieve China Mobile data, one would simply listen to TopicA .

2. Partitions

A partition is a directory on a broker server; each topic may have multiple partitions, and each partition’s data is kept in .log files, similar to database partitions for performance.

Multiple partitions enable parallel processing, which greatly improves throughput.

Analogy: topics correspond to HBase tables, partitions correspond to HBase regions; blocks in HDFS allow large files to be distributed across many servers.

Notes:

Partitions have replicas to avoid single‑point‑failure.

Partition numbers start from 0.

Producers write data to partitions.

3. Producers and Consumers

Producers send data into Kafka; consumers read data from Kafka.

4. Kafka Cluster Architecture

A topic can be created with three partitions, each stored on a different broker. Topics are logical; partitions are the physical storage units.

Versions prior to 0.8 lacked a replica mechanism, causing data loss on broker failures.

Replica : each partition can have multiple replicas. One replica is elected as the leader , the others are followers . Producers write to the leader; followers sync from the leader; consumers read from the leader.

5. Consumer Groups

The group.id property defines a consumer group. Example configuration: conf.setProperty("group.id","tellYourDream") Only one consumer in the same group can consume a given partition. Different groups can each consume the same topic independently.

consumerA:
    group.id = a
consumerB:
    group.id = a
consumerC:
    group.id = b
consumerD:
    group.id = b

Thus, consumer group a can have multiple consumers processing different partitions in parallel without overlapping messages.

consumer group:a
    consumerA
    consumerB
    consumerC

6. Controller and ZooKeeper Coordination

Kafka relies on a controller node (master) that works with a ZooKeeper ensemble. All brokers register themselves in ZooKeeper under /brokers/. The first broker to register becomes the controller.

The controller watches ZooKeeper directories, generates cluster metadata (e.g., partition assignments, replica locations) and distributes this metadata to all brokers.

When a new topic is created (e.g., /topics/TopicA), the controller detects the change, creates the partition layout in ZooKeeper, and informs followers to create the corresponding directories and log files.

7. Performance Features

Sequential writes : Kafka appends records to the end of log files, avoiding costly random‑write seeks and achieving near‑memory speed on modern disks.

Zero‑copy : Kafka uses Linux sendFile (NIO) to transfer data directly from disk to the network socket, eliminating extra user‑space copies.

Log segment storage : Each partition’s log is split into segments (default max 1 GB). When a segment fills, a new segment is created (log rolling). Example segment files:

00000000000000000000.index
00000000000000000000.log
00000000000000000000.timeindex
...
00000000000005367851.index
00000000000005367851.log
00000000000005367851.timeindex
...

The numeric prefix represents the starting offset of that segment.

Network design : Clients first connect to an Acceptor , which forwards requests to a pool of processor threads (default 3). Processors enqueue requests into a thread pool (default 8 threads) that handles reads/writes. This three‑layer reactor model can be tuned by increasing the number of processors or thread‑pool size to improve throughput.

8. Additional Resources

The article concludes with an invitation to join a technical learning community for further discussion.