What Is a Kafka Consumer Group Coordinator?

What Is a Coordinator

Coordinator is a role that coordinates multiple consumers to work correctly, e.g., computing partition assignment strategy or handling join/leave logic, similar to Kafka's controller.

Coordinator Roles

There are two kinds: Consumer Group Coordinator and Consumer Coordinator.

Consumer Group Coordinator

The GroupCoordinator acts as a central processor for all consumer coordinators. Every consumer interaction goes through it.

Elect a leader consumer client.

Handle join‑group requests.

Synchronize new assignment after rebalance.

Maintain heartbeat with clients.

Manage committed offsets stored in __consumer_offset.

Consumer Coordinator

Each client has a consumer coordinator that sends requests to the group coordinator and processes callbacks.

Send join‑group request.

Send sync‑group request (leader also computes assignment data).

Send leave‑group request.

Maintain heartbeat thread.

Send offset‑commit request.

Coordinator Operation Flow

Group Coordinator Selection Logic

Kafka creates a GroupCoordinator for each partition of the internal topic __consumer_offset. The number of partitions (default 50) determines the number of coordinators. A group is mapped to a partition by hash(group.id) % partitionCount. The leader replica of that partition hosts the coordinator.

JoinGroup Process

When a client starts or reconnects, it issues a JoinGroup request.

JoinGroup sequence diagram

Client sends first request, coordinator returns a MemberId.

Client sends second request with the MemberId.

GroupCoordinator builds MemberMetadata and caches it.

Coordinator moves group state to PreparingRebalance .

Initializes Generation data; state becomes CompletingRebalance or Empty if no members.

Assembles JoinGroupResult and returns to members; leader also receives all members' metadata.

Leader sends SyncGroupRequest to compute new partition assignment; other members follow.

SyncGroup Process

After JoinGroup, each client sends a SyncGroup request to obtain the new assignment. The leader computes the assignment and the coordinator propagates it to all clients.

LeaveGroup Process

When a client shuts down or fails, it triggers a LeaveGroup request. The coordinator monitors heartbeats; on failure it ejects the client and triggers a rebalance.

Heartbeat Detection

Clients maintain a heartbeat thread after joining a group. The coordinator checks each client's heartbeat and removes expired members, causing a rebalance.

Partition Assignment Strategy

The group must use a single assignment strategy. The coordinator selects the strategy supported by all members, preferring the one that appears earlier in each member's partition.assignment.strategy list.

Example cases:

Case‑1: supported strategies are roundrobin and rang. Votes give roundrobin two votes, so it is chosen.

Case‑2: only rang is common, so rang is chosen.

If no common strategy exists, the coordinator throws INCONSISTENT_GROUP_PROTOCOL exception.

[2022-09-08 14:34:12,508] INFO [Consumer clientId=client2, groupId=consumer0] Rebalance failed. (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
org.apache.kafka.common.errors.InconsistentGroupProtocolException: The group member's supported protocols are incompatible with those of existing members or first group member tried to join with empty protocol type or empty protocol list.
[2022-09-08 14:34:12,511] ERROR Error processing message, terminating consumer process:  (kafka.tools.ConsoleConsumer$)

Q&A

1. Which strategy is used when members configure different strategies? All members must share the same strategy; the coordinator follows the selection logic described above.

2. How do other consumers know a message has been consumed? After consumption, the offset is committed to the internal topic __consumer_offset. The key format is group+topic+partition, and the topic uses compact retention.

3. What is the offset storage structure? The key is group+topic+partition; the value contains the offset and metadata (see diagrams).

4. How to reset an offset? Send a new value for the same key to __consumer_offset or a tombstone message to delete it.

5. Will expanding __consumer_offset lose offsets? (Not answered in the source.)