From Auto to Manual: Mastering Consumer Offsets for Ten‑Million QPS Systems

Introduction

A real payment loss incident revealed that the default automatic offset commit in Kafka silently advanced the committed offset while messages were still in memory, leading to lost or duplicated messages when the consumer crashed.

What Offsets Do

Four offset concepts are defined:

Production offset : the latest message position in a partition.

Fetch offset : the next position the consumer requests from the broker.

Processing offset : the position up to which the consumer's business logic has actually processed messages.

Committed offset : the position reported to the broker for resume after a crash.

In ideal cases these offsets are equal, but with automatic commits the committed offset often exceeds the processing offset, creating a window where messages can be lost.

Automatic Commit: The Sweet Trap

Most MQ clients enable automatic commit by default (e.g., Kafka's enable.auto.commit=true with auto.commit.interval.ms=5000). This works for low‑traffic scenarios but hides two critical semantics:

Trap 1: Message Loss

When a crash occurs within the window between committed and processed offsets, the unprocessed messages are skipped. The system shows no errors, no lag alerts, and normal consumption rates, yet business data diverges.

Trap 2: Duplicate Consumption

Reversing the timeline, automatic commits can cause the same messages to be consumed twice. In a test, 99 messages were processed twice, leading to duplicate charges, coupons, or SMS if the business logic is not idempotent.

Automatic commit essentially bets that processing speed matches fetch speed and that crashes never happen at the critical window; this bet is acceptable at 100 K QPS but disastrous at 10 M QPS.

Two Core Principles for Manual Commit

Principle 1: Commit only after successful processing. If processing fails, do not commit, so the offset stays and the message is retried.

Principle 2: Business idempotency is a prerequisite for manual commit because network glitches, commit timeouts, or forced kills can otherwise cause duplicates.

Following these principles, the consumption loop becomes the diagram shown below (image omitted for brevity). A dead‑letter path is required for messages that cannot be processed after retries.

Sync vs. Async Commit

Manual commit can be synchronous or asynchronous, each with trade‑offs. Pure synchronous commit at 10 M QPS is unacceptable due to RTT overhead, while pure async risks silently losing messages if a failed commit is overwritten by a later successful one.

The industry practice is “async‑first, sync‑fallback”: batch commits are sent asynchronously for throughput, and a synchronous commit is performed before rebalance, consumer shutdown, or at critical batch boundaries.

Delivery Semantics and Offsets

The three classic delivery semantics map to offset behavior:

At‑most‑once : commit before processing; may lose messages but never duplicate.

At‑least‑once : process before commit; may duplicate but never lose (default for 99 % of workloads).

Exactly‑once : processing and commit must be atomic; requires transactional support in the producer and coordinated business transactions.

Exactly‑once cannot be guaranteed by the MQ alone when external stores are involved; the consumer must ensure atomicity via its own transaction or idempotent logic.

Transactional Binding of Business and Offset

To approach exactly‑once, bind business results and offset writes to the same storage using its transaction capability. A common pattern is for the consumer to maintain its own offset table, read the last offset on startup, seek to it, and then process and commit together, eliminating the inconsistency window.

This approach is standard for CDC, reconciliation, and billing scenarios but introduces the need for partitioned offset tables, cleanup policies, and increased database load.

Batch Commit and Frequency

Committing every message at 10 M QPS is infeasible; batch commits are the norm. Three batch strategies exist, with the most popular being a hybrid: commit when either 1 000 messages are processed or 200 ms have elapsed, whichever comes first.

A critical detail: if a batch contains a failed message, the commit must stop at the last successfully processed offset, not the end of the batch, to avoid acknowledging unprocessed messages.

Sequential vs. Concurrent Consumption Offsets

Increasing parallelism introduces out‑of‑order commits. For example, processing 1 000 messages across 16 workers may finish offset 200 before offset 50, leading to lost messages if the highest offset is committed.

The correct approach is to track the minimum unacknowledged offset and only advance the committed offset when all earlier messages are successfully processed, preventing “rewind” losses.

This may cause slow messages to block batch progress, so slow‑message timeouts and dead‑letter routing are needed.

Offset Strategy Evolution Across Traffic Levels

A roadmap shows how offset strategies evolve from automatic commit at low traffic (10 K QPS) to manual, idempotent, dead‑letter‑aware designs at high traffic (10 M QPS). The evolution is driven by matching consistency requirements to traffic, not by a single “best” solution.

Often the offset strategy lags behind traffic growth, causing teams to encounter the “sleep‑depriving” failures described at the article’s start.

Conclusion

Automatic offset commit offers convenience but trades consistency for ease. At scale, you must reclaim control: implement manual commits with processing‑first semantics, async‑first sync‑fallback, idempotent business logic, dead‑letter handling, batch windows, and proper monitoring. Doing so turns a hidden risk into an engineered, reliable component of a ten‑million‑QPS system.