Understanding Kafka: Core Concepts, Architecture, and Performance Secrets

Kafka Basics

Kafka serves as a message system that acts like a warehouse, providing caching and decoupling between producers and consumers. It stores data on disk rather than in memory, enabling reliable persistence.

1. Topic (主题)

A topic in Kafka is analogous to a table in a relational database; it is a logical grouping of messages.

To consume data from a specific source, a consumer simply subscribes to the relevant topic (e.g., TopicA).

2. Partition (分区)

Each topic is divided into multiple partitions, which are stored as directories on different brokers. Partitions improve performance by allowing parallel processing across threads, similar to HBase’s table/region design.

Partitions are replicated for fault tolerance; each replica can be a leader or follower. The leader handles writes from producers, while followers synchronize from the leader.

3. Producer (生产者)

Producers send messages to Kafka topics.

4. Consumer (消费者)

Consumers read messages from Kafka topics.

5. Message (消息)

The unit of data stored in Kafka is called a message.

Kafka Cluster Architecture

A topic with three partitions can be distributed across three brokers. Each partition can have multiple replicas; one replica is elected as the leader, and the others act as followers.

Replica (副本)

Replicas ensure data safety. Typically, two replicas per partition are sufficient.

Consumer Group (消费者组)

Consumers belong to a consumer group identified by group.id. Within a group, only one consumer reads from a given partition, preventing duplicate consumption. conf.setProperty("group.id","tellYourDream") Different groups can consume the same topic independently.

consumerA:
    group.id = a
consumerB:
    group.id = a
consumerC:
    group.id = b
consumerD:
    group.id = b

Controller

The controller is the master node that coordinates the cluster via Zookeeper. It monitors broker registrations, elects leaders, and distributes metadata.

Zookeeper Coordination

All brokers register themselves in Zookeeper under /brokers/. The controller watches these nodes, builds the cluster metadata, and propagates it to all brokers.

Performance Optimizations

Sequential Writes

Kafka writes data sequentially to disk, achieving near‑memory speeds because sequential disk writes are much faster than random writes.

Zero‑Copy

Kafka uses Linux’s sendFile (NIO) to transfer data directly from disk to the network socket, eliminating extra memory copies and context switches.

Log Segmentation

Each partition’s log files are limited to 1 GB, making it easier to load segments into memory for processing.

00000000000000000000.index
00000000000000000000.log
00000000000000000000.timeindex

00000000000005367851.index
00000000000005367851.log
00000000000005367851.timeindex

00000000000009936472.index
00000000000009936472.log
00000000000009936472.timeindex

Network Design

Client requests first hit an Acceptor, which forwards them to a pool of processor threads (default three). Processors place requests into a queue, which is handled by a thread pool (default eight threads) that reads, writes, and responds to client operations, forming a three‑layer network architecture.