Why Does Kafka Mark a Healthy Consumer as Dead and Force a Rebalance?

Why a Consumer Can Appear Alive Yet Be Declared Dead

In early Kafka versions the consumer ran on a single thread that both fetched records, executed business logic, and sent heartbeat messages. If business logic blocks (e.g., a database timeout or a full GC), the thread cannot send heartbeats in time. The broker, not receiving a heartbeat, assumes the client has crashed and immediately initiates a rebalance.

Dual‑Thread Architecture

Modern Kafka separates these responsibilities into two threads:

Heartbeat thread sends heartbeats at a fixed interval. The broker treats the consumer as alive as long as it receives heartbeats within session.timeout.ms (default 45 seconds).

Business thread repeatedly calls poll() to fetch records and execute business logic. Its liveness threshold is max.poll.interval.ms (default 5 minutes).

When the business thread is blocked longer than max.poll.interval.ms, the broker marks the consumer as “fake dead”: it still receives heartbeats, but it has not fetched new data, causing potential data backlog. The broker then forces the consumer out of the group, triggering a rebalance.

Typical Conflict Scenario

Assume a consumer pulls 500 messages in one batch. Because downstream database writes are slow, processing those 500 messages takes 6 minutes. During this time the business thread cannot return to the poll() loop, so the interval between successive poll() calls exceeds the 5‑minute default. The broker interprets this as loss of consumption capability and initiates a rebalance.

How to Avoid “Fake Dead”

1. Reduce the Pull Batch Size

Configure the maximum number of records per poll() call: max.poll.records: 50 Splitting a large batch into many small, frequent batches ensures that even if each record takes 100 ms, processing 50 records only consumes about 5 seconds, allowing the thread to return to poll() quickly and stay within the timeout.

2. Increase the Poll Interval

For worst‑case scenarios (downstream service outage, network jitter), raise max.poll.interval.ms: max.poll.interval.ms: 600000 This should be tuned together with max.poll.records; setting it too high can delay detection of a truly dead consumer, causing partitions to be reassigned after a long pause.

3. Asynchronous Multithreaded Consumption

If individual business operations are inherently long‑running, offload them to a custom Java thread pool. The main thread only fetches records and immediately hands them to worker threads, returning to poll() within seconds.

Two risks arise:

Auto‑commit loss : With auto‑commit enabled, the next poll() may commit offsets for records that are still being processed in the pool. If the consumer crashes, those in‑flight messages are lost.

Manual‑commit disorder : Off‑loading introduces out‑of‑order processing. Manually committing offsets from worker threads can cause duplicate consumption unless a sliding‑window commit mechanism or per‑partition queue is implemented, which adds considerable complexity.

Final Thoughts

Distributed systems define liveness in two ways: process liveness (process up, port open, heartbeats alive) and business liveness (the ability to process incoming data). When writing consumer logic, evaluate both max.poll.records and max.poll.interval.ms to avoid unnecessary rebalances caused by prolonged processing times.