Skip to content
Kafka Producer Internals: Batching, Compression, Idempotence & Transactions Deep Dive

Kafka Producer Internals: Batching, Compression, Idempotence & Transactions Deep Dive

This article dissects Kafka producer internals through real incidents, covering async send mechanics, memory management, idempotence via PID/epoch/sequence numbers, transaction semantics, configuration trade-offs, source code walkthroughs of doSend/append/Sender/idempotence validation, and 20 interview Q&As with practical tuning guidelines.

Architecture Pony
Architecture Pony
Architecture Pony

The previous article covered Kafka's skeleton: partitions, replicas, ISR, offsets. But when writing business code you face a different beast — a send() method hiding seven parameters, two threads, and three timeout constants. Defaults work in test but clash in production: retries enabled without idempotence, oversized batches exhausting memory, zero linger.ms halving throughput. This article explains exactly what decisions happen from send() to broker disk.

Three core threads: ① send() is asynchronous — it only enqueues to memory and returns; real network I/O runs on a separate thread. ② All reliability rests on idempotent sequence numbers , not on retries. ③ Throughput, latency, and memory are three allocations from the same pool; changing one inevitably moves the other two.
Incident 1: Retries duplicated orders. Payment callback producer used acks=all , retries=3 , but idempotence off. Network blip: broker wrote successfully, response lost on return. Client retried — same message appeared twice in partition. Downstream accounting summed by message count, over-counted 300 orders, 120k discrepancy. Post-mortem key insight: acks=all guarantees "written", not "exactly once". The gap is exactly the idempotence switch and a PID.
Incident 2: Business threads stuck in send() . Default buffer.memory=32MB . Sender backed up due to a slow broker, pool exhausted. Every subsequent send() blocked on memory allocation, up to max.block.ms=60s . 200-thread pool fully occupied in 3 minutes, QPS dropped from 8000 to 200. Not Kafka being slow — backpressure propagated up the synchronous call chain because an async client was used synchronously.
Incident 3: Custom partitioner scrambled order status. Team used Math.abs(key.hashCode()) % partitions . Worked until partitions expanded from 24 to 48. Same order key mapped to different partitions under old vs new mapping. Consumer used "per-partition local cache aggregation"; two messages for same order landed in two instances' caches, state machines overwrote each other, orders stuck at "paid not shipped". Official partitioner uses murmur2 — same remapping on expansion. So "same key same partition" only holds while partition count is stable.

1. Technical Analysis: Three Components and Three Constants

1.1 A Message's Three-Stage Journey

KafkaProducer

has only two threads with clean separation. main thread handles all business-side work: interceptors, key/value serialization, partition selection, pushing a record into the partition's queue, then returns immediately — never does network I/O. Sender thread is a background daemon looping three tasks: pull ready batches, group by broker, send and handle responses. The only handoff point is the RecordAccumulator — a deque array organized by partition.

This structure dictates three things. First, send() latency is unrelated to network; it only depends on "can I grab a 16KB memory chunk", so when it's slow look at buffer not broker. Second, batching happens inside the accumulator — multiple messages for same partition append to the same ProducerBatch until batch.size or linger.ms triggers handoff to Sender. Third, retries also happen on Sender side — business thread sees only a Future; whether it retries 3 or 30 times, main thread continues its work.

Figure 1: Producer three-stage: app thread → accumulator → Sender, two thread boundaries
Figure 1: Producer three-stage: app thread → accumulator → Sender, two thread boundaries

One detail: Future.get() turns async back into sync. Common pattern: send().get() looks like "confirm delivery" but actually serializes the entire send path (queue + batch + network round-trip + retries) onto current thread. Load test QPS stalls, RT jitters — 80% this pattern. Correct approach: use callbacks ( Callback) or collect futures and batch get() at the end.

1.2 Memory Ledger: How buffer.memory Gets Consumed

buffer.memory

is a global pool (default 32MB), shared by all partitions . Not partitioned; allocated in "chunks": each partition needing data takes a chunk equal to batch.size (default 16KB), wrapped as a ProducerBatch. Subsequent messages for same partition append into that chunk until full 16KB or linger.ms elapsed, then sealed, handed to Sender, returned to pool after broker ack.

Three numbers simultaneously decide "is pool enough": partitions × batches-per-partition × batch.size, plus in-flight batches awaiting ack. Example: 100 partitions, 2 batches each, 32KB batch = 6.4MB in queues, plus in-flight — 32MB easily exhausted. Once pool empty, send() blocks — note: blocks, not drops; waits up to max.block.ms (default 60s), then throws TimeoutException: Failed to allocate memory.

Figure 2: Memory ledger: buffer.memory sliced into batch chunks, pool empty blocks
Figure 2: Memory ledger: buffer.memory sliced into batch chunks, pool empty blocks

Counter-intuitive point: Increasing buffer.memory may not help; decreasing might be right . If downstream consumption slow causes broker throttling (quotas), larger pool just increases "how many messages stacked in memory", latency and memory rise together, eventually OOM. Correct direction: expose backpressure earlier and more controllably — smaller buffer, max.block.ms set to hundreds of ms, let send fail fast, business degrades — far better than blocking 60s.

1.3 Three Time Constants: Nesting Relationships

Producer has five timeout params but only their nesting containment matters: delivery.timeout.ms is total budget (default 120s), must be ≥ linger.ms + request.timeout.ms; request.timeout.ms is "single request wait for response" ceiling (default 30s); retries and retry.backoff.ms decide how many retries fit in total budget; max.block.ms is independent line governing only "allocate memory / fetch metadata" blocking.

Figure 3: Three time constants: total budget, single request, and retry scheduling
Figure 3: Three time constants: total budget, single request, and retry scheduling

Common config errors: setting delivery.timeout.ms smaller than request.timeout.ms (client throws ConfigException on start); only tuning request.timeout.ms without delivery.timeout.ms — 120s budget compresses 30 retries into 4. Don't compute retries from retries value; use delivery.timeout / (request.timeout + retry.backoff) — that's the real retry count. 1.3s RTT, 30s request timeout, 120s budget → max ~3 retries, completely different from retries=2147483647 "infinite retries".

1.4 Partitioner: Key Decides Everything

Partitioner answers one question: which partition for this message. Has key → murmur2(key) mod partition count — same key always same partition, the sole source of Kafka ordering. No key → version dependent: pre-2.4 round-robin; 2.4+ sticky — fill a batch then switch partition , significantly improving batch fill rate, reducing small batches and request count.

Figure 4: Partitioner: keyed hash vs keyless sticky, and hot-key long tail
Figure 4: Partitioner: keyed hash vs keyless sticky, and hot-key long tail

Two traps. First, fixed key : some teams send all messages with same key (e.g. constant "order"), all land in one partition, 23 of 24 partitions idle, consumer limited to one consumer — throughput capped. Second, partition expansion : key-to-partition mapping depends on total partition count; 24→48 changes mapping for most keys, all "partition-local ordering" logic breaks. To expand, either use "partition mapping table + dual-write transition" or make consumer idempotent per key — don't bet on mapping stability.

1.5 Idempotence: PID, Epoch, Sequence Number

Idempotence implementation is simple: client startup requests a PID (Producer ID) from broker; every message carries triple — PID, epoch, sequence number . Sequence number per partition starts at 0, increments by 1. Broker side each partition has a ProducerStateManager recording "last sequence number written for this PID on this partition". On receive: equals expected → normal write; less than expected → duplicate from retry, discard and return success ; greater than expected → gap (message lost), throw OutOfOrderSequenceException.

Figure 5: Idempotent deduplication: how triple makes retries safe, and broker's three judgments
Figure 5: Idempotent deduplication: how triple makes retries safe, and broker's three judgments

Idempotence brings two hard bindings many miss: enabling enable.idempotence=true forces acks=all, retries=Integer.MAX_VALUE, max.in.flight.requests.per.connection≤5. First two for safety; third to guarantee "retry request order not messed" — 5 is Kafka-verified safe upper bound (broker caches at most 5 batches' state). So you don't set these three; if you do they're overridden or conflict.

Idempotence boundary must be remembered : only guarantees "single producer session, single partition" no duplicates. Producer restart, PID re-assigned (e.g. idle beyond transactional.id expiry), or switch partition — sequence resets, dedup chain broken. Cross-partition atomic write needs transactions; cross-session dedup needs business unique key — two separate fences, not interchangeable.

1.6 Transactions: Cross-Partition Atomic Write

Transactions add three things on top of idempotence: a transactional.id for cross-session PID recovery and epoch bump (old session writes auto-invalidated); a TransactionCoordinator recording transaction state (stored in __transaction_state, 50 partitions); and two-phase commit — write prepare to coordinator, coordinator writes transaction markers (commit/abort) to all involved partitions, then visible to consumers.

Consumer must cooperate: set isolation.level=read_committed, consumer sees only "committed" messages, uncommitted blocked by LSO (Last Stable Offset). Cost is latency — under read_committed consumer must wait for transaction end; a long transaction stalls entire partition consumption. So don't put slow ops (e.g. sync external calls) inside transaction; transaction.timeout.ms default 60s, expiry triggers coordinator abort.

1.7 Interceptors: Only Client-Side Point to Mutate Message Body

Interceptors have two methods, one in one out: onSend called before message enters accumulator — can modify key, value, add headers, cannot change topic or partition (partition already decided); onAcknowledgement called on broker ack or failure — used for success-rate and end-to-end latency stats. It's the only client extension point that can "touch message body", so many infra capabilities hook here: auto-inject traceId, gray-release tagging, per-message serialized size stats, large-message detection and alerting.

Critical discipline: onSend runs on business thread — one remote call here adds RTT to every send; onAcknowledgement runs on Sender thread — one sync log write here adds disk I/O to entire send pipeline. Both must be pure in-memory lightweight compute; any DB write or RPC must be offloaded to async queue. Worst production case: interceptor queried Redis for blacklist check, 2ms RT × 8000/sec saturated business thread pool.

2. Production Practice: Two Complete Configs and Parameter Interplay

2.1 Config 1: Financial Grade (Idempotence + Transaction + No Loss No Duplicate)

Scenario: payment ledger, requires "no loss, no duplicate, auditable", QPS ~3000, latency tolerance 50ms. Config focus isn't maxing params but making all reliability-related defaults explicit to prevent version upgrades or colleague mistakes.

# Payment ledger producer: no loss no duplicate, latency acceptable
bootstrap.servers=k0:9092,k1:9092,k2:9092
acks=all
enable.idempotence=true
transactional.id=pay-tx-prod-01
max.in.flight.requests.per.connection=5
delivery.timeout.ms=120000
request.timeout.ms=30000
retry.backoff.ms=200
linger.ms=5
batch.size=32768
buffer.memory=67108864
compression.type=zstd
max.request.size=1048576
# Serializer package: org.apache.kafka.common.serialization
key.serializer=...StringSerializer
value.serializer=...StringSerializer
transaction.timeout.ms=60000
# Interceptor uniformly injects traceId for reconciliation
interceptor.classes=com.acme.TraceInterceptor

Three subtle but critical settings: retry.backoff.ms=200 (default 100ms retries too aggressively during broker jitter, can knock down just-recovered node); linger.ms=5 (5ms wait lifts batch fill from 3 to 40 messages, P99 only +5ms, classic good trade); buffer.memory=64MB (with 3000/sec × 1KB ≈ 3MB/s write rate, sustains 20s backlog, leaves headroom for max.block.ms degradation).

2.2 Config 2: Log Grade (High Throughput, Tolerate Tiny Loss)

Scenario: behavior tracking, 80B events/day, 400 bytes each, can lose last few hundred ms on node crash. Config converts every penny to throughput: large batch, long linger, lz4, idempotence off .

# Tracking log producer: throughput first
bootstrap.servers=k0:9092,k1:9092,k2:9092
acks=1
enable.idempotence=false
linger.ms=50
batch.size=262144
buffer.memory=268435456
compression.type=lz4
max.in.flight.requests.per.connection=5
max.request.size=4194304
delivery.timeout.ms=180000
request.timeout.ms=60000
retries=3
# Client-side rate limit to avoid crushing shared cluster
client.id=tracking-producer-07

Gains are stark: batch from 16KB to 256KB, linger 0→50ms, lz4 compression ratio 3-4x , single-node throughput from 120k/sec to 450k/sec, network bandwidth actually dropped 60%. Cost: last 50ms of data may still be in memory unsent, process kill loses it — acceptable for tracking, absolutely not for payments. Same cluster, same API, difference is business pricing of "lose one message" .

2.3 Parameter Interplay: Three Ceilings Determine Real Throughput

Don't tune by feel; compute three ceilings, minimum is your roof. Ceiling 1: memory — buffer.memory / (partitions × batch.size) = "how many batches can queue simultaneously"; < 2 means frequent blocking. Ceiling 2: connection — per broker connection, max.in.flight × batch.size / RTT = connection throughput ceiling; 10GbE RTT 0.5ms, 5 in-flight, 32KB batch → ~320MB/s, far above single-disk. Ceiling 3: batching window — larger linger.ms fuller batches but higher tail latency; pick "smallest linger that fills batch", usually 5-20ms.

One combo that must move together: max.request.size ≥ batch.size × 1.2 , else single batch plus protocol header exceeds request limit, client throws RecordTooLargeException. Similarly broker message.max.bytes and topic max.message.bytes must be raised together — any one of the three unchanged, large messages stall at different checkpoints — and error locations differ, easy to fix one and think solved.

2.4 Seven Pre-Launch Validations

Config written must be verified, else all params just "look right". ① Idempotence really on? Use AdminClient or logs confirm no ConfigException, acks forced to all. ② Duplicates actually deduplicated? Write script that intentionally resends same message twice; check broker logs for DUPLICATE_SEQUENCE, partition count only +1. ③ Out-of-order possible? Set max.in.flight=5, enable idempotence, load test with incrementing sequence numbers, consumer verifies strict increment. ④ Memory exhaustion? Push 1.5x target QPS, watch waiting-threads metric — sustained >0 means trouble. ⑤ Timeout chain self-consistent? Artificially slow broker response 5s, observe client retry vs timeout, validate total budget. ⑥ Partition skew? Print per-partition message/byte counts; max/mean ratio should be <1.3. ⑦ Compression effective? Compare record-send-rate vs outgoing-byte-rate; ratio should match configured compression ratio.

Most skipped: #2 and #6. Former is sole proof of reliability — idempotence switch state cannot be proven by "I configured it", only by "duplicates were eaten" ; latter is common root cause of performance incidents — skew >3x, consumer scaling useless because hot partition can only have one consumer.

2.5 Post-Mortem: How Those 300 Duplicates Were Found

Break incident into timeline because "how to investigate" is more reusable than "what's the conclusion". 02:14 reconciliation alert, payment ledger vs channel bill diff 120k; 02:30 check consumer, __consumer_offsets offsets continuous no rollback, rule out duplicate consumption; 03:00 pull producer logs, group by traceId, find same traceId hit two messages with adjacent offsets — proves broker actually wrote two, not consumer read twice ; 03:20 inspect client config, enable.idempotence explicitly set false, comment "disabled for throughput during load test"; 03:40 locate 02:05-02:12 broker jitter, client retried 3 times.

Three questions worth noting. Why retry causes duplicate? Because first request's response lost on return, client cannot distinguish "not written" from "written but ack lost", so retries. Why no error in logs? Because retry succeeded, client thinks all good, only DEBUG logs "Retrying" — production log level INFO, nothing visible. Why only 300 not all? Because jitter lasted 7 minutes, only batches to one affected broker, others normal. Explains why such accidents are "small volume but hard to detect".

Fix in three layers: Layer 1 immediately enable idempotence (verify acks forced to all); Layer 2 add business unique key , use serial number as DB unique index, duplicate insert conflicts, discard and audit; Layer 3 augment monitoring , in client interceptor stats callback exception types, also emit "retry succeeded" — sudden retry rate spike is failure precursor even if no loss this time .

2.6 Instance Count and Thread Count: More Isn't Always Better

KafkaProducer

is thread-safe, one instance can be shared by all application threads , internally fixed two threads (main + Sender) plus a cleaner thread. So "create multiple producers for concurrency" is classic misunderstanding — multiple instances don't raise single-node throughput, instead lower batch fill rate: 10 instances each sending 1/10 volume, each batch never fills, request count 10x, compression ratio worse.

When to use multiple instances? Only two cases: 1. Isolation — same app sends high-priority payments and low-priority tracking, two instances with different client.id, acks, quota, no interference; 2. Single instance memory insufficient — buffer.memory already 256MB still frequently exhausted, backlog genuinely huge, splitting instances = splitting pool, each independent backpressure. Beyond that, one JVM one instance, multi-thread shared is standard. Monitoring note: multi-instance requires unique readable client.id (business name + sequence), else metrics merge, impossible to locate which instance retries.

3. Core Source Code: Four Key Functions

3.1 doSend : Message's Path Into Memory

KafkaProducer.send

is a shell; real logic in doSend, fixed four steps: interceptor → serialization → fetch metadata and select partition → append to accumulator. Third step can block — if client lacks topic metadata, triggers fetch here, waits up to max.block.ms. That's why "first send unusually slow".

// KafkaProducer.doSend (simplified)
ProducerRecord<K,V> intercepted =
interceptor.onSend(record);
byte[] keyBytes = keySerializer.serialize(
record.topic(), record.key());
byte[] valBytes = valueSerializer.serialize(
record.topic(), record.value());
// Metadata missing blocks here max.block.ms
Cluster cluster = metadata.fetch();
int partition = partitioner.partition(
record.topic(), record.key(),
keyBytes, valBytes, cluster);
RecordAccumulator.RecordAppendResult res =
accumulator.append(record.topic(),
partition, timestamp, keyBytes,
valBytes, headers, callback,
maxBlockTimeMs);
// Return Future, actual send in Sender thread
return new FutureRecordMetadata(
res.future, ...);

3.2 append : Contend Memory, Cut Batches, Blocking Point

append

is the only place that truly blocks business thread. Logic: take last unsealed batch from partition's deque tail, try append; if batch full (remaining space insufficient) or already sealed, request new memory from BufferPool — if unavailable, spin-wait here until a batch is acked and returned or timeout. Two easily misunderstood points.

// RecordAccumulator.append (simplified)
Deque<ProducerBatch> dq =
getOrCreateDeque(tp);
synchronized (dq) {
if (closed) throw new IllegalState();
Deque<ProducerBatch> deque = dq;
ProducerBatch last = deque.peekLast();
if (last != null) {
FutureRecordMetadata f =
last.tryAppend(args);
if (f != null) return new AppendResult(
f, false, false);
}
// Doesn't fit: allocate new memory, may block
int size = Math.max(batchSize, ...);
BufferPool pool = free;
ByteBuffer buf = pool.allocate(
size, maxBlockTimeMs);
...
}

First point: batch cut is by bytes, not count . A 20KB message won't fit 16KB batch, gets a larger chunk allocated on demand (Kafka allocates > batch.size), called "large message degradation" — one 1MB message occupies 1MB pool, few such messages drain buffer. Second point: lock here is partition-level lock (synchronized on deque), not global, so high concurrency different partitions don't block each other; but BufferPool allocation is global — pool empty, all partitions wait together, worst case.

3.3 Sender : One Loop, Three Checks

Sender's run is a while-true loop, each round does three things: pull ready batches (filled batch.size or waited linger.ms, or node needs heartbeat), group by broker (batches for same broker across partitions merged into one request), send . Before send, a key check: in-flight request count on that connection vs limit; if at limit, skip this round, wait next.

// Sender.sendProducerData (simplified)
for (Node node : readyNodes) {
int inflight = inFlightRequests
.count(node.idString());
if (inflight >= maxInFlight) {
// Node reached in-flight limit, skip this round
continue;
}
List<ProducerBatch> batches =
accumulator.drain(node, ready);
if (batches.isEmpty()) continue;
sendProduceRequest(node, batches);
}

In-flight limit is the source of reordering : if batch 1 fails retry, batch 2 already succeeded, broker sees 2 then 1. Setting max.in.flight=1 preserves order but throughput cut to 1/5; so since 0.11 official solution is enable idempotence, let broker use sequence numbers to reject out-of-order writes (receives seq=1 but already wrote seq=2, rejects and forces client resend), rather than client-side serialization.

3.4 Idempotence Validation: Broker's Three Judgments

Broker side each partition maintains a ProducerStateManager, for each active PID stores a ProducerState: first/last sequence numbers and timestamps of last five batches. Only five because >5 means client already messed up, better reject and force session rebuild.

// ProducerStateManager (simplified)
ProducerState state =
producers.get(producerId);
int expected = state.lastSeq + 1;
if (seq == expected) {
state.update(seq, offset); // normal write
} else if (seq < expected) {
// Duplicate: return success, don't write log
return DUPLICATE_SEQUENCE;
} else {
// Gap: client lost messages
throw new OutOfOrderSequenceException(
"expected " + expected + " got " + seq);
}

These three lines' practical significance far exceeds complexity: they turn "retry" from "may duplicate" into "definitely idempotent" . Also explains why idempotence sets retries to infinite — since duplicates are eaten, no need to limit count; only boundary is delivery.timeout.ms. So in idempotent scenario tuning focus isn't retries, but whether total budget can survive longest outage .

4. Advanced: Seven Accounts

4.1 Ordering Account: Three Conditions Must Hold Simultaneously

To guarantee "same business entity messages ordered", need three things together: same key, same partition, producer-side no reorder . First two via partitioner; third via max.in.flight=1 or idempotence. Consumer adds one more: single partition consumed by only one consumer in group (Kafka mechanism, naturally true), but if you use thread pool to process partition messages in parallel, order breaks again — most common "configured key but still out-of-order" cause.

4.2 Idempotence Failure Account: Three Chain-Break Scenarios

Idempotence not permanent; three situations reset sequence: producer restart (new PID), PID expiry (broker producer.id.expiration.ms default 7 days, but idle sessions may be cleaned earlier), transaction timeout or epoch bump . During break, retries may duplicate. So financial correct posture is idempotence + business unique key double insurance : message carries business serial, consumer uses DB unique index or Redis SETNX as safety net, duplicates discarded and logged.

4.3 Latency Account: Linger Buys Throughput, Pays Tail Latency

linger.ms

0→10, P50 latency barely moves, P99 stably increases ~10ms , in exchange request count drops an order of magnitude. Trade worth it in most scenarios because Kafka latency bottleneck usually not client batching but broker acks=all round-trip. Exception: low-traffic topics . Few messages/sec, batch never fills, linger is pure wait — such topics should set linger 0, batch small, don't copy high-traffic config.

4.4 Compression Account: Choosing Among Four Algorithms

Compression happens client-side, decompression broker and consumer, compresses batch not single message — another reason to batch. Selection by scenario: gzip highest ratio but heavy CPU, high latency, suits cold data archival; snappy fast but mediocre ratio, being phased out; lz4 best throughput balance (Kafka official benchmarks show lz4 highest throughput); zstd ratio near gzip, speed near lz4, available 2.1+, suits "bandwidth is bottleneck" cross-datacenter.

4.5 Large Message Account: Four Gates Interlock

A 5MB message passes four gates: client max.request.size, broker message.max.bytes, topic max.message.bytes, consumer fetch.max.bytes. All four must be raised, and causes chain reactions : large message monopolizes batch memory, lowers compression ratio, slows replica sync. Correct practice: don't send large messages — store big objects in object storage (OSS/S3), message carries only reference and digest, consumer fetches on demand. This is hard rule in nearly all big companies.

4.6 Observability Account: Five Must-Watch Metrics

Client metrics expose problems earlier than broker: record-queue-time-avg (time in accumulator, rising means send can't keep up with produce); batch-size-avg (actual batch size, far below config means low traffic or keys too scattered); buffer-exhausted-rate (pool exhaustion count, >0 means must expand or degrade); waiting-threads (blocked business threads, direct incident precursor); record-error-rate and request-latency-avg (distinguish network vs broker slowness). These five on one dashboard, producer-side incidents detectable ~5 minutes early.

4.7 Version Account: Three Must-Know Changes

2.4 introduced sticky partitioner , keyless batch fill rate greatly improved, default partitioner changed from round-robin; 3.0+ enable.idempotence defaults true , meaning acks forced to all — many "upgraded and got slower" root cause here; 4.0 removed ZooKeeper , also dropped deprecated params (e.g. ZK connect strings), upgrade must clean config files or startup fails.

5. High-Frequency Interview Questions (with Answers)

20 Q&A covering: send() blocking points, idempotence implementation, idempotence vs acks config, idempotence boundaries, ordering guarantees, max.in.flight effects, linger.ms vs batch.size relationship, delivery.timeout.ms calculation, buffer exhaustion causes, buffer.memory increase pros/cons, partitioner selection, partition expansion impact, transaction vs idempotence difference, read_committed cost, compression layer placement, large message handling, key producer metrics, Future.get() pitfalls, 3.0 upgrade slowdown reason, producer memory sizing. Each answer concise and technically precise.

6. One-Page Cheat Sheet

Two threads: main does serialization & partition selection (no network), Sender does batching, send, retry. One handoff: RecordAccumulator, partition-organized deque array. Three seal conditions: batch.size full, linger.ms elapsed, Sender poll. Three timeouts: max.block.ms governs blocking, request.timeout.ms governs single request, delivery.timeout.ms governs total budget (≥ linger + request). Idempotence triple: PID + epoch + partition sequence, broker three judgments (normal / duplicate discard / out-of-order exception). Ordering four conditions: same key, same partition, idempotence or no retry, consumer single-threaded. Tuning mnemonic: throughput → batch & linger; latency → linger & acks; stability → buffer & max.block; no-loss-no-duplicate → idempotence + business unique key.

7. Summary & Next Episode Preview

Producer complexity all stems from "async" induced misalignment : you think send done means arrived, actually it's in memory waiting for batch; you think retry 3 times enough, actually total budget only covers 1.5; you think acks=all is reliable, actually it only guarantees written, not exactly once. Three sentences to remember this article: send slow → check buffer and metadata first, not broker ; reliability rests on idempotent sequence numbers, not retries ; throughput, latency, memory are three allocations from same pool, change one inevitably moves the other two .

Next episode: Consumer & Consumer Group. Will cover three rebalance triggers, cooperative vs eager protocol differences, three offset commit timings, and "consumption stalled but process healthy" type hardest-to-debug issues. Includes complete consumer config and full investigation of a lag spike to 8 million.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

configurationPerformance Tuninginterviewidempotencetransactionsproducercompressionsource codebatching
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.