Why Kafka Offsets That Appear Committed Still Cause Duplicate Consumption After Restart

Understand the Timing of Automatic Offset Commit

Most projects use Kafka's default auto‑commit settings:

enable.auto.commit=true
auto.commit.interval.ms=5000  # default: commit every 5 seconds

Auto‑commit does not happen after each record; it occurs during the next poll() call, committing the highest offset retrieved in the previous poll. If the consumer processes messages up to offset 350 but the last auto‑commit was at offset 200, offsets 200‑350 remain uncommitted.

Rebalance Is the Real Killer

In ideal cases (graceful shutdown), the issue is limited. In production, a rebalance often triggers duplicate consumption. A consumer is considered dead and a rebalance is triggered when:

Heartbeat timeout (default session.timeout.ms = 45 s) occurs.

The interval between two poll() calls exceeds max.poll.interval.ms (default 5 min).

The consumer voluntarily leaves the group via close() or process termination.

The second case is the most hidden and common. If a batch of messages is fetched and processing is slow (e.g., HTTP calls or bulk DB writes), the consumer may have processed offsets 1000‑1100 but only committed offset 1000. After rebalance, a new consumer starts from offset 1000, causing duplicates.

More Hidden Pitfall: GC Causing Poll Interval Timeout

A full GC pause can last seconds or minutes, freezing all threads. During this time, the consumer cannot call poll(). When the pause ends, the gap between two poll() calls exceeds max.poll.interval.ms, instantly triggering a rebalance.

The logs often only show a CommitFailedException, making the problem hard to spot.

Manual Commit Is Not Foolproof Either

Switching to manual commits (e.g., consumer.commitSync();) moves the responsibility to the application, but the window between processing a record and committing its offset still exists. If a rebalance, network glitch, or crash occurs in that window, the offset is lost.

Achieving exactly‑once semantics requires transactional commits or idempotent processing on the consumer side.

Practical Solutions

Solution 1: Reduce Batch Size and Commit Promptly

# Pull fewer records each poll for faster processing
max.poll.records=50
# Extend poll interval to avoid premature rebalance
max.poll.interval.ms=600000
# Disable auto‑commit
enable.auto.commit=false

while (true) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(1000));
    for (ConsumerRecord<String, String> record : records) {
        processMessage(record);
    }
    // Commit after processing the batch
    consumer.commitSync();
}

Solution 2: Per‑Record Commit (Higher Overhead, Highest Accuracy)

for (ConsumerRecord<String, String> record : records) {
    processMessage(record);
    consumer.commitSync(Collections.singletonMap(
        new TopicPartition(record.topic(), record.partition()),
        new OffsetAndMetadata(record.offset() + 1)
    ));
}

Solution 3: Idempotent Consumer Logic (Recommended)

public void processMessage(ConsumerRecord<String, String> record) {
    String bizId = extractBizId(record);
    // Use a unique business ID to ensure single processing
    if (redis.setIfAbsent("consumed:" + bizId, "1", 24, TimeUnit.HOURS)) {
        doBusinessLogic(record);
    } else {
        log.warn("Duplicate consumption, skipping: {}", bizId);
    }
}

Even with timely commits, extreme scenarios like network partitions or process crashes can still cause duplicates, so business logic must be able to handle them.

Takeaway

Kafka’s offset mechanism guarantees at‑least‑once delivery, not exactly‑once. The gap between processing a message and persisting its offset can be exploited by rebalance, GC pauses, or crashes, leading to duplicate consumption. Therefore, do not rely solely on offsets; implement idempotent processing or transactional commits to achieve true exactly‑once semantics.