Why Does Kafka Re‑Consume Tens of Thousands of Messages After a Successful Offset Commit?

Understanding the timing of automatic commits

Most projects use Kafka's default auto‑commit settings:

enable.auto.commit=true
auto.commit.interval.ms=5000  # default 5 seconds

Automatic commit does not happen after each message; it occurs every 5 seconds at the next poll() call, committing the highest offset retrieved in the previous poll.

Thus offset 200 may be committed while processing has already reached 350; offsets 200‑350 remain uncommitted.

Rebalance is the real culprit

In an ideal graceful shutdown the problem ends, but in production a rebalance often causes duplicate consumption.

Kafka marks a consumer dead and triggers a rebalance in three cases:

Heartbeat timeout (default session.timeout.ms = 45 seconds) without a heartbeat.

Poll interval exceeding max.poll.interval.ms (default 5 minutes).

Consumer voluntarily leaves the group via close() or process shutdown.

The second case is the most hidden and common. Example scenario: the consumer pulls a batch, processing is slow (e.g., HTTP calls or bulk DB writes). Offsets 1000‑1100 are processed, but only offset 1000 was committed. After rebalance a new consumer starts from 1000, causing duplicates.

More hidden pitfall: GC causing poll interval timeout

A full GC pause can freeze all threads for seconds or minutes, making the interval between two poll() calls exceed max.poll.interval.ms, which directly triggers a rebalance. The log may only show a CommitFailedException without obvious clues.

Manual commit is not foolproof

Switching to manual commit (e.g., consumer.commitSync()) gives the application control over when to commit, but the window between processing a message and committing its offset still exists. If a rebalance, network glitch, or process crash occurs in that window, the offset can be lost.

Achieving exactly‑once semantics requires either transactional commits or idempotent processing on the consumer side.

Practical solutions

Solution 1: Reduce batch size and commit promptly

# Pull fewer records each poll, process faster
max.poll.records=50
# Extend poll interval
max.poll.interval.ms=600000
# Disable auto commit
enable.auto.commit=false

while (true) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(1000));
    for (ConsumerRecord<String, String> record : records) {
        processMessage(record);
    }
    // Commit after processing the batch
    consumer.commitSync();
}

Solution 2: Commit per record (slightly slower but most precise)

for (ConsumerRecord<String, String> record : records) {
    processMessage(record);
    // Commit offset of the processed record
    consumer.commitSync(Collections.singletonMap(
        new TopicPartition(record.topic(), record.partition()),
        new OffsetAndMetadata(record.offset() + 1)
    ));
}

Solution 3: Idempotent consumption (recommended)

Even with timely commits, network partitions or crashes can still cause duplicates, so critical business logic should be idempotent. Example using Redis SETNX to ensure a business ID is processed only once.

public void processMessage(ConsumerRecord<String, String> record) {
    String bizId = extractBizId(record);
    if (redis.setIfAbsent("consumed:" + bizId, "1", 24, TimeUnit.HOURS)) {
        // First time processing
        doBusinessLogic(record);
    } else {
        // Duplicate, skip
        log.warn("Duplicate consumption, skip: {}", bizId);
    }
}

Takeaway

In a normal restart, the 5‑second auto‑commit interval may leave the last batch uncommitted; slow processing triggers rebalance, causing another consumer to start from an earlier offset; GC pauses can also cause poll‑timeout rebalance. Manual commit still leaves a window where offsets can be lost.

Kafka's offset mechanism guarantees at‑least‑once delivery, not exactly‑once. Therefore, consumer code should not rely solely on offset to prevent duplicates; business logic must be able to handle repeated processing.