Why Kafka Processed My Order Twice: Offset Commit Timing & Idempotency Fixes

The incident began when an inventory service processed the same order message twice, 34 seconds apart, with no retry logic or consumer restart. Logs showed two identical deductions for order SO20261026001 at 09:12:03 and 09:12:37. Investigation of the consumer group offsets revealed the root cause: after the first processing, the committed offset remained at the batch start; a rebalance 34 seconds later reassigned the partition to another instance, which read from the committed offset and reprocessed the batch.

Three Offset Positions on the Consumer Side

A single partition maintains three positions simultaneously, and confusing them leads to incorrect explanations of duplicate consumption:

position – the next offset to fetch on the next poll(); exists only in client memory and is lost on restart.

committed offset – the offset stored in the internal __consumer_offsets topic; used as the starting point after restart or rebalance.

log end offset – the partition's latest write position; determines how many messages remain unread.

During normal consumption, position advances continuously, while committed offset catches up according to the commit strategy. The gap between them equals the number of messages that will be redelivered if the process crashes or the partition is reassigned. In auto-commit mode this gap can span an entire batch; in manual acknowledgment it shrinks to the processing time of a single message.

# View consumer group offsets and lag
kafka-consumer-groups.sh --bootstrap-server broker:9092 \
  --describe --group inventory-service

TOPIC      PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG  CONSUMER-ID
order-topic  0         180420          180915          495  consumer-1-9f2a
order-topic  1         220118          220118            0  consumer-1-9f2a

CURRENT-OFFSET

is the committed offset. LAG (495 on partition 0) represents messages fetched but not yet committed – the exact quantity that would be reprocessed after a crash.

Auto-Commit and poll() Advancement

Kafka clients default to enable.auto.commit=true with auto.commit.interval.ms=5000. The name suggests commits happen after processing, but actually the commit timer is checked inside poll(). When the interval expires, the current position (which already points to the end of the fetched batch) is committed. Business code has only just started processing the first record. Thus 500 records are fetched, position jumps to 500, and five seconds later offset 500 is committed while processing may be at record 12. The timer knows nothing about business progress.

If a crash occurs in this window, the entire batch (records 12–500) is redelivered. No messages are lost because committed offset never exceeds processed position, but the duplicate volume is uncontrolled. With default max.poll.records=500 and 5-second commit interval, a 10 ms per-record processing keeps up, but 100 ms per record causes significant divergence. In streaming workloads this can overwhelm downstream deduplication tables.

AckMode and Commit Timing in Spring Kafka

Since Spring Kafka 2.3, using @KafkaListener forces enable.auto.commit=false unless explicitly overridden in the consumer factory, because the container provides its own commit mechanism. The commit timing is then governed by AckMode (default BATCH): after all records from a single poll() are processed, one commit is issued. This favors throughput but causes the whole batch to replay if any record fails.

spring:
  kafka:
    consumer:
      enable-auto-commit: false
    listener:
      # BATCH commits once after the whole batch; failure replays entire batch
      ack-mode: batch
      concurrency: 3

For per-record control, switch to MANUAL or MANUAL_IMMEDIATE and add an Acknowledgment parameter to the listener. The difference lies in immediacy: MANUAL – marks the offset in memory; actual commit occurs on the next poll(). MANUAL_IMMEDIATE – performs a synchronous commit immediately upon acknowledge() invocation.

@KafkaListener(topics = "order-topic", groupId = "inventory-service",
               containerFactory = "manualAckFactory")
public void onOrder(ConsumerRecord<String, String> record, Acknowledgment ack) {
    // Persist first, then acknowledge – reversing order equals auto-commit
    inventoryService.reserve(record.value());
    // Under MANUAL_IMMEDIATE this blocks until broker acknowledges
    ack.acknowledge();
}

The placement of acknowledge() defines the duplicate window. Calling it before business logic yields at-most-once semantics (messages lost on crash). Calling it after shrinks the window to a single record's processing time, at the cost of an extra synchronous round-trip per record.

Under MANUAL, acknowledge() does not commit immediately; it only marks. If the process dies mid-batch, even marked offsets are not committed, and the whole batch replays. For true immediacy, MANUAL_IMMEDIATE is required, accepting the throughput penalty.

Batch listeners can use nack(index, delayMs) (Spring Kafka 2.3+) to commit successful records up to the failure point and replay only the failed record and subsequent ones. nack() must be called from the consumer thread, requires batch listener and AckMode=MANUAL, and behaves differently under transactions.

@KafkaListener(topics = "order-topic", containerFactory = "batchAckFactory")
public void onBatch(List<ConsumerRecord<String, String>> records, Acknowledgment ack) {
    for (int i = 0; i < records.size(); i++) {
        if (!inventoryService.tryReserve(records.get(i).value())) {
            // Record i fails: commit previous, replay from i (inclusive)
            // Second param = wait milliseconds before redelivery
            ack.nack(i, 1000L);
            return;
        }
    }
    ack.acknowledge();
}

Rebalance and Partition Ownership

Partitions change owners during rebalance, triggered by consumer join/leave, topic partition changes, or consumer deemed lost. Two timeouts govern loss detection: session.timeout.ms (default 45s since Kafka 3.0, previously 10s) – monitors heartbeat arrival. Heartbeats are sent by a background thread independent of business logic. max.poll.interval.ms (default 300s) – measures time between two poll() calls. Slow processing, blocked queries, or thread-pool queuing delay poll() while heartbeats continue; exceeding this timeout ejects the consumer. The two timeouts cover distinct failure modes.

Upon ejection, partitions are immediately reassigned. The new owner reads from the committed offset. The original consumer may still be processing its batch; when it later calls acknowledge(), a CommitFailedException is thrown because it is no longer in the group. This exception signals that duplication is already happening – two instances hold the same batch, and whichever writes to the database first wins.

2026-10-26 09:12:33.114 WARN  [order-consumer-1] ... member consumer-1-9f2a has left group due to consumer poll timeout has expired
2026-10-26 09:12:33.690 INFO  [order-consumer-1] ... Revoking previously assigned partitions [order-topic-0]
2026-10-26 09:12:34.028 INFO  [order-consumer-2] ... Setting newly assigned partitions [order-topic-0]
2026-10-26 09:12:37.902 INFO  [order-consumer-1] c.e.c.InventoryListener : 扣减库存 orderNo=SO20261026001 qty=2

The third and fourth lines show the partition reassigned at 34s, yet consumer-1 continues processing and writes the duplicate at 37.9s because the code did not verify partition ownership.

Kafka 3.0+ uses the incremental cooperative rebalance ( CooperativeStickyAssignor), transferring partitions one by one without a full-group pause. This reduces idle time but does not eliminate the duplicate window during handover; both old and new instances may read the same records. Offset commits thus affect fetch rhythm, partition ownership, and business idempotency – a misconfiguration in any of these manifests as duplicate processing, and logs alone cannot reveal which write succeeded first or why the second delivery occurred.

max.poll.interval.ms and Processing Latency

In the opening incident, max.poll.interval.ms was set to 30,000 ms (copied from a template for lightweight consumers). A batch of 500 records at 80 ms each requires 40 seconds. At the 375th record (30 seconds), the poll timeout fired, triggering rebalance. The duplicate log appeared 34 seconds after the first, matching the rebalance round-trip plus the new consumer's fetch time.

Two corrective directions: configuration and code. Configuration: raise max.poll.interval.ms to twice the worst-case batch duration, and reduce max.poll.records to 100 to bound batch processing under 8 seconds.

spring:
  kafka:
    consumer:
      max-poll-records: 100
      properties:
        # 2x worst-case batch time, headroom for GC and network jitter
        max.poll.interval.ms: 60000
        session.timeout.ms: 45000
    listener:
      ack-mode: manual_immediate

Lowering max.poll.records increases fetch requests and slightly reduces throughput, but makes processing time predictable. In slow-consumer scenarios, the cost of a rebalance (duplicates and pause) far outweighs a few extra fetches.

Code side: instrument per-record latency to find the slowest records. An 80 ms average may hide a few 20 ms records and several 140 ms records caused by multiple database queries or external calls. Average latency is misleading; percentile logging identifies the exact hotspots.

Idempotent Consumption Landing Point

All previous measures only shrink the duplicate window; they cannot eliminate it. Network partitions, coordinator failover, or process kills can still cause redelivery. Kafka's idempotent producer only deduplicates sender-side retries, not consumer-side redelivery. Business correctness requires idempotency in the consumption logic.

Use the message's business unique key as the idempotency key: orderNo for orders, channel transaction ID for payment callbacks. Never use offset – offsets change after rebalance, causing replays to be treated as new messages.

A deduplication table with a unique index is the lowest-cost approach. The database enforces uniqueness; application-level check-then-insert races under concurrency. The DDL below includes a unique key on biz_key and a timestamp for periodic cleanup.

CREATE TABLE t_consume_log (
  id           BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
  biz_key      VARCHAR(64)  NOT NULL COMMENT '业务幂等键，如渠道流水号',
  topic        VARCHAR(64)  NOT NULL,
  consumed_at  DATETIME(3)  NOT NULL,
  PRIMARY KEY (id),
  -- Unique index is the sole guarantee; app-level check-then-insert leaks under concurrency
  UNIQUE KEY uk_biz_key (biz_key)
) ENGINE = InnoDB DEFAULT CHARSET = utf8mb4
  COMMENT = '消费去重表，按月清理';

Insert with INSERT IGNORE or catch duplicate-key exception (MySQL 1062). Zero affected rows means the unique index blocked the duplicate; return immediately without executing business logic.

@Transactional
public void reserve(OrderMsg msg) {
    int inserted = consumeLogMapper.insertIgnore(msg.getOrderNo(), "order-topic");
    if (inserted == 0) {
        // Unique index intercepted – this message was already processed
        log.info("重复消息已跳过 orderNo={}", msg.getOrderNo());
        return;
    }
    // Deduplication record and business write in same transaction
    // avoids half-state where deduplication succeeds but business fails
    inventoryMapper.deduct(msg.getSku(), msg.getQty());
}

The deduplication table grows indefinitely; partition by month or schedule cleanup of records older than the maximum replay window. Kafka's topic retention is 7 days; keeping deduplication records for one month is sufficient. The window must not be too short – deleting early removes the deduplication capability.

Redis can serve as a fast-path filter using setIfAbsent to reduce database load, but it cannot be the sole guarantee – Redis restarts or replication failovers can lose writes. The database unique index remains the ultimate safety net. Finally, idempotency must cover every side effect in the message: if inventory deduction is deduplicated but SMS or downstream MQ messages are not, users still receive duplicate notifications. Downstream actions must either carry the same idempotency key or be driven by state changes.