Comprehensive Guide to Kafka Architecture, Messaging Mechanisms, Replication, Controllers, and Consumer Rebalance

Kafka is a mainstream distributed messaging engine and stream‑processing platform, often used as an enterprise message bus and real‑time data pipeline. This article selects several core topics to help readers quickly master Kafka, covering its architecture, message‑sending mechanism, replication, controller, and consumer rebalance.

Kafka architecture Kafka message‑sending mechanism Kafka replication mechanism Kafka controller Kafka rebalance mechanism

1. Kafka Quick Start

Kafka is a distributed messaging engine and stream‑processing platform, frequently used as an enterprise message bus, real‑time data pipeline, and sometimes even as a storage system. Early versions positioned Kafka as a high‑throughput distributed messaging system; today it has evolved into a mature distributed messaging engine and streaming platform.

1.1 Kafka Architecture

Kafka follows the producer‑consumer model: producers send messages to a specific partition of a topic on a broker, and consumers pull messages from one or more partitions for processing. The topology is shown below:

Kafka relies on Zookeeper for distributed coordination, storing and managing metadata such as broker information, topic definitions, partition and replica assignments.

1.2 Kafka Terminology

Producer: the message creation and sending side.

Broker: a Kafka instance; multiple brokers form a Kafka cluster, typically one broker per machine.

Consumer: the side that pulls messages for consumption. A topic can have many consumers; a group of consumers forms a Consumer Group, and each message is consumed by only one consumer within the group.

Topic: the logical storage unit for messages on the server, usually composed of multiple partitions.

Partition: a subdivision of a topic stored across brokers to achieve load‑balanced publishing and subscribing. Each partition has multiple replicas for reliability.

Message: the actual data stored by Kafka, consisting of a key, a value, and a timestamp.

Offset: the position of a message within a partition, maintained by Kafka; consumers also store offsets to track consumption progress.

1.3 Kafka Roles and Characteristics

High throughput, low latency – Kafka can handle millions of messages per second with millisecond‑level latency.

Persistent storage – messages are persisted on disk with sequential reads/writes for performance, and replication improves reliability.

Distributed scalability – data is distributed across brokers by topic and partition, offering excellent horizontal scaling.

High fault tolerance – the cluster continues to serve requests even if a broker fails.

2. Kafka Message Sending Mechanism

The producer’s message‑sending mechanism is crucial for Kafka’s high throughput. The basic flow is illustrated below:

2.1 Asynchronous Send

Since version 0.8.2, Kafka introduced a new Producer API that sends messages asynchronously. A ProducerRecord is serialized by keySerializer and valueSerializer , routed to a partition by the partitioner, placed into the accumulator buffer, and finally sent by a dedicated Sender thread.

The buffer size is controlled by buffer.memory (default 32 MB). When the buffer is full, the producer blocks for max.block.ms before throwing a timeout exception; the size can be tuned per workload.

2.2 Batch Send

Messages in the buffer are grouped into batches. Batch size is controlled by batch.size (default 16 KB). Larger batches improve throughput, while smaller batches reduce latency.

The linger.ms parameter defines the maximum idle time for a batch; even if the batch is not full, it will be sent after this timeout.

2.3 Message Retry

The producer supports retries for transient failures (e.g., network glitches). The number of retries is set by retries (default 0). It is recommended to enable a few retries, such as 3.

3. Kafka Replication Mechanism

Each partition has a leader and one or more follower replicas. Replication (also called the Replication mechanism) is the foundation of Kafka’s high reliability and availability.

3.1 Purpose of Replication

By default a partition has one replica (controlled by default.replication.factor ). In practice a replication factor of 3 is common. Replication provides:

Redundant storage to improve data reliability.

Higher availability – if the leader fails, a follower can be elected as the new leader.

3.2 Read/Write Separation

Kafka does not support separate read and write replicas. All read/write requests go to the leader; followers only replicate data asynchronously and never serve client reads.

3.3 ISR (In‑Sync Replicas) Set

ISR is the set of replicas that are currently in sync with the leader; it always includes the leader. The ISR list is stored in Zookeeper, and only replicas in ISR are eligible for leader election.

An ISR member is considered in sync if its lag is less than replica.lag.time.max.ms (default 10 s). This design reduces the chance of data loss.

3.4 Unclean Leader Election

If ISR becomes empty (e.g., the leader crashes and no follower is in sync), Kafka can elect a non‑in‑sync replica as the new leader if the unclean.leader.election.enable flag is true. This avoids unavailability at the cost of possible data loss, illustrating the classic CAP trade‑off.

4. Kafka Controller

The controller is a core component that manages and coordinates the entire Kafka cluster with the help of Zookeeper. Any broker can become the controller, but only one broker holds this role at a time.

Zookeeper stores metadata in a hierarchical ZNode tree. Nodes can be persistent (remain after a client disconnects) or temporary (deleted when the client session ends).

The controller watches the /brokers/ids temporary nodes to detect broker additions, graceful shutdowns, or crashes.

Key responsibilities of the controller include:

Topic management – creating, deleting, and adding partitions.

Partition reassignment – executing the reassign script.

Preferred leader election – promoting the preferred replica to leader when load is unbalanced.

Cluster member management – monitoring broker lifecycle via Zookeeper watches.

Metadata service – storing the full cluster metadata and pushing updates to all brokers.

5. Kafka Consumer Rebalance Mechanism

Rebalance is the process by which all consumers in a group agree on partition assignment. During rebalance, consumers pause consumption, which can severely affect throughput.

5.1 When Rebalance Occurs

Change in the number of consumers within a group.

Change in the number of topics subscribed by the group.

Change in the number of partitions of a subscribed topic.

5.2 Kafka Coordinators

Kafka introduces two coordinators:

Group Coordinator – runs on a broker and handles consumer group registration, offset storage, and rebalance coordination.

Consumer Coordinator – runs inside each consumer client to maintain heartbeats and communicate with the Group Coordinator.

5.3 How to Avoid Unnecessary Rebalance

Consumers send periodic heartbeats (default every 3 s) to the Group Coordinator. If heartbeats stop, the coordinator assumes the consumer is dead and triggers a rebalance. Two important parameters are:

session.timeout.ms – the time the coordinator waits before considering a consumer dead.

max.poll.interval.ms – the maximum interval between two poll() calls (default 5 min). Exceeding this also causes the consumer to leave the group.

Additional recommendations:

Configure session.timeout.ms and heartbeat.interval.ms appropriately (larger session timeout reduces rebalance frequency).

Adjust max.poll.interval.ms based on business logic to avoid long processing pauses.

Monitor GC activity; frequent Full GC can pause consumer threads and trigger rebalance.

6. Summary

This article covered Kafka’s architecture, message‑sending pipeline, replication mechanism, controller role, and consumer rebalance details. After reading, you should have a solid grasp of Kafka’s core principles and be better prepared to work with its internals.