Understanding Apache Kafka Rebalance Protocol: Fundamentals and Practical Details

Since Apache Kafka 2.3.0, the internal rebalance protocol used by Kafka Connect and consumers has undergone several major changes.

The rebalance protocol is complex and often feels like magic; this article returns to its basics—the core of Kafka's consumption mechanism—then discusses its limitations and recent improvements.

Kafka and Rebalance Protocol 101

Back to the Basics

Apache Kafka is a distributed publish/subscribe streaming platform. Producers send messages to topics managed by a broker cluster; consumers subscribe to topics to retrieve and process those messages.

Topics are split into partitions that are distributed across many brokers. The number of partitions is defined when the topic is created and can be increased later (with caution).

Partitions are the unit of parallelism for both producers and consumers.

On the producer side, partitions enable parallel writes. When a key is used, the producer hashes the key to select a target partition, guaranteeing that all messages with the same key go to the same partition and are delivered to the consumer in order.

On the consumer side, the number of partitions limits the maximum number of active consumers in a consumer group. A consumer group groups multiple consumer clients into a logical unit for load‑balancing partitions. Kafka ensures that each partition is assigned to only one consumer in the group.

For example, a consumer group "a" with three consumers might have partitions P0→C1, P1→C2, P2→C3.

When a consumer leaves the group due to a controlled shutdown or crash, its partitions are automatically reassigned to the remaining consumers; similarly, when a consumer (re)joins, partitions are rebalanced among all members.

The ability of consumers to cooperate in a dynamic group is provided by Kafka's rebalance protocol.

Let's dive into how this protocol works.

Rebalance Protocol Overview

First, a definition of "rebalance" in the Kafka context:

Rebalance: a process where a set of distributed processes using Kafka clients and/or the Kafka coordinator form a common group and redistribute a set of resources among the group's members (source: Incremental Cooperative Rebalancing: Support and Policy).

This definition does not explicitly mention consumers or partitions; instead it uses the concepts of members and resources, because the rebalance protocol can be used to coordinate any group of processes, not just consumers.

Typical uses of the rebalance protocol include:

Confluent Schema Registry uses rebalance to elect a leader node.

Kafka Connect uses it to assign tasks and connectors among workers.

Kafka Streams uses it to assign tasks and partitions to stream instances.

The rebalance mechanism is built around two protocols: the Group Membership protocol and the Embedded Client protocol.

The Group Membership protocol coordinates members of a group; clients use a Kafka broker acting as coordinator to exchange request/response messages.

The Embedded Client protocol runs inside the client, extending the Group Membership protocol—for example, the consumer protocol that assigns topic partitions to members.

Now that we understand what the rebalance protocol is, let's see how it is implemented for partition assignment in a consumer group.

JoinGroup

When a consumer starts, it first sends a FindCoordinator request to locate the broker that coordinates its group, then initiates the rebalance by sending a JoinGroup request.

The JoinGroup request includes client configuration such as session.timeout.ms and max.poll.interval.ms ; if a member does not respond, the coordinator can eject it from the group.

It also contains two crucial fields: the list of client protocols the member supports, and metadata for executing an embedded client protocol (e.g., the partition assignment strategy). The metadata holds the list of topics the consumer subscribes to.

JoinGroup acts as a barrier: the coordinator will not send a response (or will wait for group.initial.rebalance.delay.ms ) until all consumers have sent their JoinGroup requests.

The first consumer that receives the active member list and the chosen assignment strategy becomes the group leader; other consumers receive an empty response. The leader performs the partition assignment locally.

SyncGroup

Next, all members send a SyncGroup request to the coordinator. The leader attaches the computed assignment, while the others send an empty request.

When the coordinator replies to the SyncGroup request, each consumer receives its assigned partitions, invokes the onPartitionsAssigned listener, and begins fetching messages.

Heartbeat

Periodically, each consumer sends a Heartbeat request to the coordinator to keep its session alive ( heartbeat.interval.ms ).

If a rebalance is in progress, the coordinator uses the Heartbeat response to tell the consumer to re‑join the group.

In real‑world distributed systems, failures (hardware, network, or transient errors) can also trigger rebalances.

Warnings

The first limitation of the rebalance protocol is that you cannot rebalance a single member without stopping the entire group (the "stop‑the‑world" effect).

When gracefully shutting down an instance, the consumer should send a LeaveGroup request before stopping.

Remaining consumers are notified that they must rebalance on the next heartbeat, initiating a new JoinGroup/SyncGroup round‑trip to reassign partitions.

If a consumer restarts after a short failure, it will re‑join the group, causing another rebalance and a temporary pause in data processing.

Rolling upgrades of a consumer group can be disastrous: a three‑consumer group undergoing a rolling upgrade may trigger six rebalances, significantly impacting throughput.

Common Java consumer issues include missed heartbeats due to network interruptions or long GC pauses, and not calling KafkaConsumer#poll() often enough, causing the coordinator to consider the consumer dead (session timeout) or to exceed max.poll.interval.ms .