Kafka Exactly-Once: Idempotent Producers, Transactions & Two-Phase Commit Explained
This article explains Kafka's exactly-once semantics through idempotent producers, transactions, and read-committed isolation, detailing real incidents, configuration patterns, source-code mechanisms, and the critical boundary that Kafka transactions cannot cover external side effects, requiring application-level idempotence for true end-to-end exactly-once.
This article provides a comprehensive deep dive into Kafka's exactly-once message delivery guarantees, covering the mechanisms that prevent duplicates and data loss, their boundaries, and practical configuration patterns validated by real production incidents.
Three Real-World Incidents
Incident 1: 30,000 duplicate messages causing duplicate refunds. To reduce latency, acks was changed from all to 1 and idempotence was disabled (idempotence requires acks=all). During a broker rolling restart, a network blip triggered a producer retry. The retry succeeded, but the original request had already been written. Result: 30,000 order messages consumed twice, refund API called twice. Lesson: Retries and idempotence are a pair; if you allow retries, you must enable idempotence.
Incident 2: Commit-before-process vs process-before-commit. The consumer committed offsets before processing business logic. An OOM crash caused 2,000 committed-but-unprocessed orders to be lost. Switching to process-then-commit caused the same 2,000 orders to be reprocessed on the next failure — duplicates. Commit timing is a binary choice: either risk loss or risk duplication. The real fix is business-level idempotence (unique order ID), making duplicates harmless.
Incident 3: A slow transaction freezes the entire downstream. A data team used a transaction for consume-clean-forward. A hidden 12-minute slow SQL inside the transaction prevented commit. The partition's LSO (Last Stable Offset) stalled, all read_committed consumers stopped, downstream lag grew to 1.4 million, alerts fired for 40 minutes before root cause was found. Transactions are a double-edged sword: they block dirty reads but also block all messages behind a slow transaction.
Technical Analysis: Where Duplicates and Losses Originate
A message's lifecycle has four failure points: producer retries (network blip causes resend, broker already received → duplicate), broker replica failover (leader fails before replication → loss), consumer commit timing (commit-then-process loses, process-then-commit duplicates), external side effects (message processed but DB write fails, or vice versa). Exactly-once means closing all four gaps; missing any one reverts to at-least-once.
1.1 Three Delivery Semantics Precisely Defined
Semantics are not three switches but three "promise scopes": at-most-once promises no duplicates, allows loss; at-least-once promises no loss, allows duplicates; exactly-once promises exactly one delivery. In distributed systems, exactly-once practically equals "at-least-once + deduplication". Semantics are the end-to-end product of each segment — producer exactly-once + sloppy consumer = at-least-once overall. You must first define which segment you are talking about.
1.2 Idempotent Producer: PID, Epoch, Sequence Number
Idempotence is surprisingly small: each producer instance obtains a PID (producer ID) from the broker on startup. Every batch it sends carries a monotonically increasing sequence number per partition . The broker retains the last 5 sequence numbers per partition. On receipt: if the sequence is exactly the next expected, write it; if duplicate (retry), return the previously written offset without rewriting; if a gap (out-of-order), throw OutOfOrderSequenceException. Epoch solves cross-session issues: a new instance with the same transactional.id gets a higher epoch; requests from the old instance with the old epoch are rejected ( Fenced) — the "zombie producer" fence.
Idempotence limitations: It only works for same PID, same partition retries. No atomicity across partitions (partition A succeeds, partition B fails — no rollback). No continuity across sessions (process restart → new PID, sequence resets to 0, broker cannot associate with old PID). These gaps are exactly what transactions solve. Idempotence is not a free switch: enable.idempotence=true forces acks=all, retries=MAX, max.in.flight.requests.per.connection=5 — trading throughput ceiling for correctness, a trade most businesses should make.
1.3 Transactions: Cross-Partition Atomic Writes & Two-Phase Commit
Transactions register a set of partitions as an atomic unit: producer calls beginTransaction, sends messages to multiple partitions (broker stages them invisibly), then commitTransaction. The broker-side TransactionCoordinator drives two-phase commit: first persist the transaction decision to the internal topic __transaction_state (durable "decide to commit"), then write a CommitMarker control message to each involved partition. Consumers only see messages after reading the marker. Abort writes an AbortMarker, making messages permanently invisible. The essence of two-phase commit is separating "decision" from "execution": the decision is durable, so even if the coordinator crashes mid-execution, it can complete the remaining markers on recovery, avoiding half-committed states.
When transactions coexist with normal writes, a detail matters: markers are log messages ordered by offset. In the interleaving window where a marker is written but its messages are still being appended, consumers might see partial results. Kafka uses "LSO + per-PID aggregation" to minimize this window but does not eliminate it — transaction isolation is read-committed, not serializable . Two concurrent transactions writing the same partition: first writer wins, no conflict detection. True serializability (e.g., state-machine validation) requires application-level locking or Kafka Streams' EOS single-threaded model.
1.4 read_committed and LSO: The Cost of Visibility
With isolation.level=read_committed, the broker only returns data before the LSO (Last Stable Offset) : LSO is the start offset of the first incomplete transaction . Uncommitted transaction messages and all subsequent messages are invisible — ordering guarantee prioritized, cannot skip uncommitted data to serve later messages. The cost: a single slow transaction stalls the entire partition . After transaction.timeout.ms (default 60s) the coordinator forces abort, then LSO advances. First discipline of using transactions: keep transaction granularity small, never put long-running logic between begin and commit.
Idempotence has another often-overlooked boundary: the deduplication table retains only the last 5 batches . Retries must occur within a tight window — after a producer crash/restart (PID change), the old PID's last few batches cannot be deduplicated even if retried. So "idempotence guarantees no duplicates" precisely means: retries occurring within the session lifetime and before the batch slides out of the 5-batch window . Long-running transactions, cross-restart compensations must be handled by transactions (same transactional.id with new epoch) or application layer.
1.5 Exactly-Once Boundary: Where Kafka Stops
Kafka transactions cover the "Kafka-to-Kafka" loop: consume-transform-produce (Streams' exactly-once is built this way). Once a message must produce external side effects — write MySQL, call payment API, send SMS — the transaction cannot help: commitTransaction only commits Kafka-internal visibility, it will not roll back your database. The final mile of end-to-end exactly-once is always application code : use unique key constraints for DB idempotence ( INSERT ON DUPLICATE KEY), pass idempotency keys for external calls, or use the local message table / outbox pattern to bind "write DB" and "send message" in a single local transaction.
Outbox pattern expanded: business code writes two tables in the same local DB transaction — business table and message table (outbox). The message table holds pending messages; a background poller delivers to Kafka and marks sent. "DB write succeeds but message send fails" is retried by the poller; "message sent but DB write fails" never happens — atomicity guaranteed by DB transaction, eventual consistency by polling. The message table naturally redelivers, so consumers still need idempotence — outbox eliminates loss, not duplicates; both together form the complete loop.
1.6 The Other Half of Loss: Consumer Commit Strategy
Idempotence and transactions solve "duplicates"; "loss" is solved by consumer commit timing: commit-then-process (auto-commit default) loses messages if processing fails or process crashes; process-then-commit (manual sync commit) duplicates on crash after processing but before commit. There is no third option — another CAP manifestation. Engineering conclusion: choose process-then-commit + business idempotence , making duplicates harmless; never use auto-commit + async processing (commit runs ahead of processing, loss window = entire processing queue). Streams EOS packages "process + produce + commit" into one transaction, automating the correct choice.
Auto-commit has a hidden trap: enable.auto.commit=true commits every 5 seconds in the poll loop, committing the offset returned by the last poll — not the offset of fully processed records. If processing is async (poll hands off to thread pool), commits always race ahead of processing, loss window = entire processing queue . This is the #1 root cause of "Kafka occasionally loses messages" — silent, no errors, no retries, only discovered during reconciliation.
1.7 Parameter Tiers: Three Steps from "No Loss" to "Exactly Once"
Reliability builds in strict order; skipping steps causes failures: Tier 1 "No Loss" — acks=all + min.insync.replicas=2 + unclean.leader.election.enable=false, with consumer process-then-commit; Tier 2 "Kafka-Internal No Duplicates" — enable idempotence; for cross-partition atomicity or cross-session continuity add transactions + read_committed; Tier 3 "End-to-End Exactly Once" — all external side effects made idempotent. Each tier builds on the previous: chasing "no duplicates" before achieving "no loss" is like putting a precise lid on a leaky bucket.
Production Practice: Full Config & Postmortem
2.1 Config 1: Producer Full Reliability (Idempotence + Transactions)
For pipeline services (consume → transform → produce), transactions enabled, idempotence auto-activated. Transactional ID named "service-name + instance-group"; all instances in the same group share the same transactional.id (fencing distinguishes old vs new instances via epoch).
# ==== producer full reliability config ====
# Pipeline service: consume -> transform -> produce
# --- Base reliability ---
acks=all
enable.idempotence=true
# Idempotence auto-locks: retries=MAX, in-flight<=5
max.in.flight.requests.per.connection=5
# --- Transactions ---
# Shared by same-group instances, fencing via epoch
transactional.id=order-sync-pipeline
# Transaction timeout: default 60s, set per longest processing time
# Too long stalls LSO, too short kills normal business
transaction.timeout.ms=30000
# Concurrent transactions: >= business thread count
transactional.id.expiration.ms=86400000
# --- Transaction usage (code skeleton) ---
# After configuring tid in props:
producer.initTransactions()
while (running) {
records = consumer.poll(...)
producer.beginTransaction()
for (r : records)
producer.send(transform(r))
// Commit consumer offsets into the same transaction
producer.sendOffsetsToTransaction(
offsets, consumer.groupMetadata())
producer.commitTransaction()
}2.2 Config 2: Broker & Consumer + Verification Scripts
Broker secures the "no loss" baseline, consumer picks the right isolation level, plus verification tooling — reliability must be verified, otherwise it's just a feeling .
# ==== Broker side (reliability trio) ====
default.replication.factor=3
min.insync.replicas=2
# Prefer partition unavailability over stale replica
unclean.leader.election.enable=false
# Transaction state topic also 3 replicas
transaction.state.log.replication.factor=3
transaction.state.log.min.isr=2
# ==== Consumer ====
# Read only committed, avoid aborted messages
isolation.level=read_committed
# Disable auto-commit: manual sync commit after processing
enable.auto.commit=false
# Max poll interval, prevent slow processing triggering rebalance
max.poll.interval.ms=300000
# ==== Verification scripts: duplicate & loss probes ====
BS=localhost:9092
# 1) Duplicate detection: consume with dedup counting
kafka-console-consumer.sh \
--bootstrap-server $BS --topic orders \
--group audit-g1 --from-beginning \
--property print.key=true \
| sort | uniq -c \
| awk '$1>1{c++} END{print c" dup"}'
# 2) Loss detection: produce with unique sequence, consumer
# checks sequence continuity (gap = loss)
# 3) Transaction state inspection
kafka-transactions.sh \
--bootstrap-server $BS list
kafka-transactions.sh describe \
--bootstrap-server $BS <transactionalId>Script design notes: duplicate probe uses uniq -c to find keys appearing more than once, requires messages carry business unique key (order ID, event ID); loss probe uses continuous sequence numbers — producer assigns seq during load test, consumer asserts no gaps. Both require dedicated test topics — mixed with production traffic, "duplicates" and "replays" are indistinguishable. Recommend baking these probes into CI as chaos tests, run on every version upgrade.
2.3 Postmortem: The 40-Minute LSO Stall
Back to Incident 3: initial reaction to lag spike was insufficient consumer capacity, added two consumer instances — no effect, because consumers weren't "processing slowly" but "waiting for LSO to advance". Diagnosis: check if consumer group position stalls while new messages keep arriving, then run kafka-transactions list to find long-running Prepare state transactions. Fix in three layers: kill slow transaction to force timeout abort; reduce transaction granularity from "whole batch" to "per-record forward"; reduce transaction.timeout.ms from default 60s to 30s. Afterwards, similar LSO alerts dropped from 3/month to 0. Root cause was not Kafka, but placing long-running operations inside the transaction boundary — this rule is now in the team's code review checklist.
2.4 Reliability Acceptance Checklist: Six Gates Before Launch
Unverified config = no config. Six gates, each must pass: ① Duplicate probe — produce with unique sequence, consumer asserts zero duplicates; ② Loss probe — sequence continuity check, kill -9 producer and broker each once; ③ Fencing drill — two instances same transactional.id, verify old instance fenced; ④ LSO drill — artificially create slow transaction, confirm timeout abort restores consumption; ⑤ Crash recovery — kill -9 consumer, replay, verify business idempotency key absorbs duplicates; ⑥ Reconciliation loop — daily T+1 reconciliation as final safety net, discrepancies traced to specific component. All six pass before claiming exactly-once.
2.5 Canary & Rollback: How to Deploy Reliability Changes
Reliability config changes have asymmetric risk: downgrade (e.g., revert acks to 1) is silent, surfaces days later in reconciliation; upgrade (enable idempotence/transactions) is loud, misconfig fails fast. Change order: upgrade first, downgrade later; small traffic first . New transactional service runs on shadow topic for a week, then cuts real traffic. Downgrade (removing idempotence for performance) requires review and dedicated reconciliation on downgrade day. Rollback plan must be rehearsed: rolling back transactional service to non-transactional version requires read_committed consumers to downgrade simultaneously, otherwise they won't see in-flight messages — producer and consumer reliability tiers must move together .
Core Source Code: Four Supporting Mechanisms
Four code sections map to four key points: idempotent batch retry deduplication, transaction state machine, coordinator commit progression, LSO visibility calculation. Exception branches simplified, main flow retained.
3.1 Sender: Idempotent Batch Retry & Deduplication
// Sender: idempotent mode retries by seq
void addToInflightBatches(...) {
// Retried batch keeps original sequence
// -> broker deduplicates by seq, no duplicate write
inFlightBatches.add(batch);
}
// Broker-side ProducerStateManager:
boolean canWrite(sequence) {
// Consecutive -> normal write
// Duplicate -> return original offset (idempotent)
// Gap -> OutOfOrderSequenceException
return seq == lastSeq + 1 || isDup(seq);
}Note: retried batch's sequence stays unchanged — that's the whole secret of "retry without duplicate": broker matches same seq, returns previous offset. max.in.flight=5 ensures the reordering detection window is sufficient; larger would bypass the window.
3.2 TransactionManager: State Machine Driven
// TransactionManager: state machine core
enum State {
UNINITIALIZED, READY, IN_TRANSACTION,
COMMITTING_TRANSACTION,
ABORTING_TRANSACTION, ...
}
// Every API call validates state transition:
void transitionTo(State target) {
if (!isTransitionValid(current, target))
throw new IllegalStateException(...);
current = target;
}
// begin/commit/abort are state transitions +
// corresponding requests enqueued, sent async by SenderTransaction API safety comes from the state machine: begin inside begin, commit after abort are intercepted. Understanding this state machine gives a map for debugging ProducerFencedException, InvalidTxnState, etc.
3.3 TransactionCoordinator: How Commit Progresses
// TransactionCoordinator: EndTxn handling
def handleEndTxn(txnId, epoch, result) = {
// 1. Validate epoch, prevent stale instance takeover
if (epoch != metadata.epoch)
throw ProducerFencedException
// 2. Persist decision to __transaction_state
// (prepare_commit / prepare_abort)
txnMetadata.state = prepareState(result)
// 3. Async write markers to all involved partitions
// all complete -> CompleteCommit
sendMarkers(txnMetadata, result)
}This is the two-phase commit embodiment: step 1 durably writes the "decision" to the transaction log (crash-recoverable), step 2 retries marker writes idempotently until all partitions complete. Orphaned prepare state transactions are cleaned up by a timeout thread, never stuck forever.
3.4 LSO Calculation: The read_committed Gate
// Partition: LSO calculation logic
def lastStableOffset(isPending: Boolean) = {
// No pending transaction -> LSO = LEO
if (!isPending) highWatermark
else
// Has pending -> start of first undecided transaction
firstUndecidedOffset
}
// Fetch for read_committed consumer:
def collectFetch(fetchOffset) = {
// Upper bound = min(LEO, LSO)
val upper = Math.min(highWatermark, lso)
log.read(fetchOffset, upper)
}LSO semantics are clear: no pending transaction → LSO = LEO, business as usual; one pending transaction → LSO stops at transaction start. This single min operation is the code-level root cause of "slow transaction stalls downstream" — also explains why shortening transaction timeout is the most effective mitigation.
Seven Accounts: Reliability Costs & Benefits
Throughput Account: Idempotence caps in-flight at 5, pipeline throughput drops 10-20%. Bought: zero duplicates on retry — for reconciliation, refund, billing, this trade is a no-brainer.
Latency Account: Transactions add 2-3 RPCs per batch (AddPartitions/EndTxn/marker), end-to-end latency up a few ms. In pipelines this isn't the bottleneck; the bottleneck is usually business logic inside the transaction boundary.
Availability Account: min.insync.replicas=2 + RF=3 tolerates 1 broker down; 2 down → partition rejects writes — by design, prefer pause over silent degradation. unclean=false same logic.
LSO Account: One pending transaction freezes the whole partition. Default 60s timeout = worst-case 60s downstream stall. Set 20-30s per business, monitor LSO divergence — most effective insurance.
External Account: Kafka transactions don't cover external systems. DB idempotence (unique key), idempotency keys for calls, outbox pattern — pick one, must land. This is the last mile of end-to-end exactly-once, and the most frequently ignored mile.
Ops Account: Two instances with same transactional.id kicking each other is a feature, not a bug, but deployment scripts must ensure rolling upgrades drain old instance first; kafka-transactions tool in runbook, LSO alert in on-call handbook.
Verification Account: Duplicate/loss/fencing/LSO drills on every major upgrade, ~half day effort. Buys confirmation that "config hasn't drifted" — config drift is the biggest silent killer of reliability.
Debugging Account: Average time to locate a "message loss" incident is 5x that of a "duplicate" incident — loss is silent (found in reconciliation), duplicates are loud (customers complain immediately). This explains why "may duplicate" is always preferable to "may lose": loud failures fix fast, silent failures are fatal.
High-Frequency Interview Questions (with Answers)
Q1: Three message semantics? Which does Kafka support? At-most-once (no dup, may lose), at-least-once (no lose, may dup), exactly-once (exactly one). Kafka supports all: no retry = at-most; retry + acks=all = at-least; idempotence + transactions + read_committed = exactly-once inside Kafka; end-to-end once needs external idempotence.
Q2: Idempotent producer principle? Trio: PID (instance identity), epoch (cross-session fencing), sequence number (per-partition incrementing batch seq). Broker keeps last 5 seqs per partition; duplicate seq returns original offset, no rewrite; gap throws exception. Only solves single-session single-partition retry duplicates.
Q3: Why does idempotence require acks=all? Idempotence deduplication relies on broker acknowledgment; acks=all ensures message written to all ISR replicas before ack, avoiding "leader acks then failover loses, retry writes different result". Enabling idempotence forces acks=all, retries=MAX, in-flight<=5; acks=1 causes startup error.</p><p><strong>Q4: What is ProducerFencedException?</strong> Two instances with same <code>transactional.id</code> run concurrently; new instance <code>initTransactions</code> gets higher epoch; old instance's subsequent requests with old epoch are rejected by broker, throwing this exception. This is the anti-zombie-producer mechanism: A presumed dead, B takes over, A revives but cannot write.</p><p><strong>Q5: Kafka transaction two-phase commit flow?</strong> Phase 1: producer calls EndTxn, TransactionCoordinator persists decision (prepare_commit/prepare_abort) to <code>__transaction_state</code>. Phase 2: Coordinator writes CommitMarker/AbortMarker to all involved partitions; when all complete, transaction finalizes. Decision persisted first, mid-crash recoverable by coordinator, no half-commit.</p><p><strong>Q6: What is __transaction_state?</strong> Internal topic storing transaction metadata: mapping tid to PID/epoch/state/partitions. Configured 3 replicas (<code>transaction.state.log.replication.factor=3</code>), Coordinator shards transactions: hash(tid) determines ownership, leader failover handled by replica group.</p><p><strong>Q7: What is CommitMarker?</strong> Control message (control batch) written to user partition, marking "PID's offset range belongs to transaction X and is committed/aborted". <code>read_committed</code> consumers filter by marker: messages before marker released or discarded per transaction result. Marker invisible to business consumers.</p><p><strong>Q8: What is LSO (Last Stable Offset)?</strong> Last stable point: start offset of first incomplete transaction. <code>read_committed</code> consumers read up to LSO — uncommitted transaction messages and all after are invisible (ordering priority). No pending transaction → LSO = LEO.</p><p><strong>Q9: read_committed vs read_uncommitted?</strong> Former reads only committed (LSO-bound), avoids dirty reads but may stall on slow transaction; latter reads all persisted data (including uncommitted, to-be-aborted), no wait but may read rolled-back data. Consistency-sensitive choose former, pure log shipping choose latter.</p><p><strong>Q10: Slow transaction impact? Prevention?</strong> LSO stalls at transaction start, whole partition freezes for <code>read_committed</code> consumers, lag spikes; at <code>transaction.timeout.ms</code> (default 60s) coordinator forces abort, recovery. Prevention: small transaction granularity (move slow logic out), shorten timeout (20-30s), monitor LSO vs lag divergence.</p><p><strong>Q11: Can Kafka transaction guarantee DB no duplicates?</strong> No. Transaction only manages Kafka-internal atomic visibility, doesn't cover external side effects. DB must self-idempotent: unique key constraint (<code>INSERT ON DUPLICATE KEY</code>), idempotency key on calls, or outbox pattern (local transaction writes business table + message table, then async send). This is the last mile of exactly-once, must be implemented in business layer.</p><p><strong>Q12: How does consume-transform-produce achieve end-to-end exactly-once?</strong> Three steps in one transaction: commit consumer offsets (<code>sendOffsetsToTransaction</code>) and output messages in same producer transaction; output idempotent (same PID same partition seq dedup); consumer uses <code>read_committed</code>. Crash → whole transaction rolls back (offsets + output together) or commits entirely — Streams EOS is exactly this.</p><p><strong>Q13: Consumer offset commit strategies & trade-offs?</strong> Commit-then-process (auto-commit): may lose — offset committed but processing incomplete on crash. Process-then-commit (manual sync): may duplicate — processing done but commit not persisted on crash, replay on restart. Engineering picks latter + business idempotence, making duplicates harmless. Auto-commit + async processing = largest loss window, forbidden.</p><p><strong>Q14: Possible message loss points?</strong> Four: producer acks=1/0 during broker failover; broker <code>unclean.leader.election.enable=true</code> elects stale replica; consumer commit-then-process; external side effect fails without rollback. Countermeasures: acks=all+min.insync, unclean=false, process-then-commit, external transaction/idempotence compensation.</p><p><strong>Q15: max.in.flight > 1 ordering guarantee? Without idempotence: in-flight>1 + retry causes reorder (batch 2 succeeds, batch 1 retries) — either in-flight=1 or sacrifice order. With idempotence: broker validates seq, gap throws exception, client with in-flight<=5 keeps window detectable — order and retry both achieved.
Q16: transactional.id design? Name by "logical service + business group" (e.g., order-sync-pipeline), same logical instance group shares one; fencing uses it to distinguish old/new instances. Note: different instance groups must not share tid (they'd kick each other); tid must be stable across restarts, no random suffix, else fencing breaks, possible double-write.
Q17: Transaction performance impact? Each transaction adds 2-3 RPCs + marker writes, ms-level; throughput impact mainly from idempotence in-flight limit and transaction serialization (same tid only one active transaction at a time). Pipeline measured 10-20% overhead, acceptable for most businesses.
Q18: How to verify exactly-once config works? Three probes: duplicate detection (consumer counts by unique key, expect 0), loss detection (producer sends continuous seq, consumer asserts 0 gaps), fencing drill (two instances same tid, old instance rejected). Plus kill -9 chaos round: producer, broker, consumer each once.
Q19: Kafka Streams EOS vs client transactions? Streams EOS packages "consumer offset commit + state store changes + downstream produce" into one producer transaction, enabled by processing.guarantee=exactly_once_v2. Internally uses same PID/epoch/transaction/marker mechanism, plus standby replicas and state changelog transactionalization.
Q20: Kafka vs RocketMQ exactly-once differences? RocketMQ 4.x native transaction messages don't provide exactly-once consumption (transaction messages solve "local transaction + send message atomicity", outbox-style), consumption semantics rely on business deduplication; Kafka since 0.11 has built-in idempotence+transactions, Streams native EOS. Flink Kafka Connector's two-phase commit depends on Kafka transaction API.
One-Page Cheat Sheet
Idempotence Trio: PID (instance identity) + epoch (cross-session fencing) + sequence (per-partition dedup). Solves single-session single-partition retry duplicates, cost: in-flight 5, throughput -10-20%.
Transaction Core: transactional.id for cross-session identity, two-phase commit (decision to __txn_state , markers to user partitions), read_committed sees only committed. LSO = first incomplete transaction start.
Consumer Binary Choice: Commit-then-process loses, process-then-commit duplicates — no third option. Choose "may duplicate", then use unique key to make duplicates harmless. Disable auto-commit + async processing.
Boundary Iron Law: Kafka transactions only cover Kafka-internal loop. DB writes, API calls, SMS all need business idempotence (unique key / idempotency key / outbox) — this is the last mile of end-to-end exactly-once.
Three Tiers: First no-loss (acks=all + min.insync=2 + unclean=false) → then no-dup (idempotence → transactions) → then end-to-end (external idempotence). Order cannot be skipped; skipping guarantees failure.
Summary & Next Episode Preview
This article compresses to one sentence: exactly-once = idempotence (single-session single-partition no-dup) + transactions (cross-session cross-partition atomic) + read_committed (consumer sees no dirty data) + business idempotence (external side-effects covered) . Four components each own a segment; missing any reverts to at-least-once; and "duplicate" is always easier to handle than "loss" — loss is money, duplicate is a reconciliation row, engineering always picks "process-then-commit + idempotency key". Parameters are just the physical form of this understanding: each of the six switches (acks, min.insync, unclean, idempotence, transactional.id, isolation.level) backs a real incident. Next episode: Monitoring & Operations — weaving all discussed metrics (lag, ISR, LSO, queue wait, request latency) into a monitoring system: what to watch, how to set thresholds, alert tiers, on-call runbooks. Starting with a "3 AM alert storm".
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
