Skip to content
Kafka Deep Dive: 5 Roles, 3 Mappings & a Message's Full Journey from Producer to Consumer

Kafka Deep Dive: 5 Roles, 3 Mappings & a Message's Full Journey from Producer to Consumer

This article dissects Kafka's core architecture through five roles, three-layer mappings, and a message's complete lifecycle, illustrating with six diagrams, two production configs, and three real incidents covering consumer stalls, disk-full broker crashes, and silent ISR shrinkage causing data loss.

Architecture Pony
Architecture Pony
Architecture Pony

Kafka handles half of real-system traffic asynchronously — order notifications, payment logs, behavioral data — yet its failure modes are subtle: consumption stalls without errors, replicas shrink silently, disks fill and kill brokers outright. This article maps Kafka's skeleton: five roles, three mappings, and one message's end-to-end journey.

Three Core Threads

Partitions are both the unit of parallelism and the unit of failure — all capacity and reliability calculations trace back to partitions.

Replica count ≠ replica health; ISR (In-Sync Replicas) is the real-time available replica count .

Consumer-group stability depends on two budgets: heartbeat budget and processing-time budget; exceed either and a rebalance triggers.

Three Real Incidents

Incident 1: Deployment Causes 40-Minute Consumption Stall

Order consumers pulled 500 records per batch, each batch taking 6 minutes due to synchronous risk-control calls — exceeding max.poll.interval.ms (default 5 min). The coordinator evicted the consumer, triggering rebalance; partitions reassigned to another instance, but offsets weren't committed, so the new instance replayed from the previous batch's start. This loop repeated for 40 minutes, lag growing from 30K to 8M, delaying downstream reconciliation. No errors logged — only "member removed due to session timeout" buried in thousands of business logs.

Incident 2: Disk Full, Entire Broker Exits

Retention set to 72 hours; normal traffic used 60% of 4 TB. A promotion tripled traffic; two data disks filled at 2 AM. Kafka's response is brutal: unwritable log directory → immediate broker shutdown , not just one partition. That broker hosted 200+ partition leaders; all triggered leader election simultaneously, client metadata refreshed en masse, upstream services spiked 5xx. Disks don't slow when full — they die, and die cleanly.

Incident 3: ISR Shrinks to 1 Undetected, 12 Minutes of Data Lost Two Days Later

A partition had RF=3. Network blip caused two followers to fall behind; ISR shrank from 3 to 1. With min.insync.replicas=1 and producer acks=1, writes continued succeeding, monitoring showed green. Two days later the sole remaining replica's disk failed; Kafka found no in-sync replicas, partition went offline. After disk replacement, the last 12 minutes of writes were gone — not delayed, nonexistent. Post-mortem: "Replica count is design value; ISR is reality value; we never monitored reality."

Technical Analysis: Five Roles & Four-Layer Structure

1.1 Five Roles: Don't Confuse Controller and Coordinator

Only five role types exist. Newcomers confuse Controller (cluster-level, single, handles partition leader election, replica assignment, broker up/down) and GroupCoordinator (group-level, each consumer group hashes to a broker partition; that broker becomes coordinator, manages members and offsets) . One-line distinction: Controller decides "which broker is leader"; Coordinator decides "which consumer eats which partition" . Troubleshooting path differs: frequent leader flips → Controller; repeated rebalances → Coordinator.

Figure 1: Kafka cluster panorama — producer → partition/replica → consumer group, offsets written to internal topic
Figure 1: Kafka cluster panorama — producer → partition/replica → consumer group, offsets written to internal topic

1.2 Topic, Partition, Replica: Three-Layer Mapping

Topic = logical category; Partition = physical log file; Replica = partition copy. A topic splits into N partitions; each partition is an independent, append-only log file sequence; each partition has R replicas spread across brokers. Partition count sets parallelism ceiling (consumers ≤ partitions); replica count sets fault tolerance (tolerates R-1 broker failures). These must be calculated separately — many incidents conflate them.

Figure 2: Partition & replica — 3 partitions, 3 replicas, leader round-robin distribution
Figure 2: Partition & replica — 3 partitions, 3 replicas, leader round-robin distribution

Leader distribution is round-robin , not stacked on one broker. If three partition leaders all sit on B0, B0 bears all write traffic; B0 fails → three partitions simultaneously unavailable. Default allocator spreads leaders and followers evenly, preferably cross-rack ( broker.rack when set). Inspection must-check: if a broker's leader count significantly exceeds average, partition allocation is skewed — usually due to post-expansion missing partition reassignment or manual topic creation with fixed brokers.

1.3 Write Path: One Message Passes Five Gates

Producer sends one message — looks like one RPC, actually five steps: ① Interceptor & serialization → ② Partitioner picks partition → ③ Accumulator batches → ④ Send to leader, append to page cache → ⑤ acks decides when to acknowledge. Step ③ is the most overlooked performance lever: client doesn't send one-by-one; batches by batch.size and linger.ms, one batch per network round-trip. linger.ms from 0 to 10 ms often doubles throughput, costing 10 ms end-to-end latency — Kafka's first high-throughput secret: trade a little latency for massive batching gain.

Figure 3: Write path — batch → leader append → follower sync → HW advances
Figure 3: Write path — batch → leader append → follower sync → HW advances

Three concepts must be separated: LEO (Log End Offset) = each replica's own write position; HW (High Watermark) = position all ISR replicas have caught up to; consumers can only read up to HW ; ISR (In-Sync Replicas) = current set of replicas keeping pace with leader. acks=all doesn't mean "all replicas written"; it means "current ISR all written" — if ISR has shrunk to 1, acks=all protects nothing. This is Incident 3's root cause.

1.4 Read Path: Why Zero-Copy Is So Fast

Read path deserves more attention — Kafka's read performance hinges on one syscall: sendfile . Traditional read-then-send needs four copies: disk → kernel page cache → user buffer → socket buffer → NIC. sendfile moves data in kernel space directly from page cache to NIC, eliminating two copies and two context switches , and completely bypasses JVM heap — why Kafka runs GB/s throughput with only 6-8 GB heap: its cache is OS page cache, not Java heap .

Figure 4: Zero-copy — sendfile vs traditional read comparison
Figure 4: Zero-copy — sendfile vs traditional read comparison

1.5 Consumer Groups & Offsets: Offset Is Also a Message

Consumer group rule: one partition consumed by at most one group member at a time; one consumer can eat multiple partitions . So consumer count > partition count → extra instances idle (common "scaling ineffective" phenomenon). Offset management is often misunderstood: committing offset = writing a message to internal topic __consumer_offsets with key = group+partition , default 50 partitions, hashed by group.id to a partition; that partition's leader broker is the group's coordinator.

Figure 5: Offset storage — __consumer_offsets key & coordinator location
Figure 5: Offset storage — __consumer_offsets key & coordinator location

Two direct consequences: ① Offset commit is a normal Kafka write — subject to acks, ISR, disk; broker hiccups slow or fail commits; ② Coordinator broker down → group's offset commits temporarily unavailable , consumers retry, possible duplicate consumption (if auto-commit enabled). Production rule: avoid large-scale rebalances at peak — during rebalance all members stop consuming and re-fetch offsets; those minutes of lag explode.

1.6 Control Plane: From ZooKeeper to KRaft

Old architecture stored metadata in ZooKeeper: brokers raced for ephemeral node /controller; winner became controller; partition leader election, replica assignment, ACLs lived in ZK. Problem: metadata existed in two places — ZK and broker memory — synced via watches; at scale (tens of thousands partitions) "controller switch → metadata out of sync" appeared, clients got stale routing, manifesting as mysterious UNKNOWN_TOPIC_OR_PARTITION.

Figure 6: Control plane — ZooKeeper vs KRaft metadata paths
Figure 6: Control plane — ZooKeeper vs KRaft metadata paths

KRaft's essence: make metadata a Kafka log too . Controllers form a Raft group; all metadata changes written as entries to __cluster_metadata; brokers pull this log like ordinary consumers. Single metadata copy, explicit ordering, replayable — controller switch from "minutes + possible inconsistency" to "seconds + strong consistency". New clusters have no reason to use ZK; 3.x migration path: ZK → KRaft dual-write → cutover; official tools exist but require maintenance window. Interview staple "What does KRaft solve?": eliminated external dependency & metadata dual-write, turned controller election from grab to Raft, pushed partition scale ceiling from tens of thousands to millions .

Core Workflows: Two Complete Production Configs

2.1 Config 1: Broker Side (KRaft, 8C32G, 10 GbE)

# /opt/kafka/config/kraft/server.properties
# 8C32G / SSD / 10GbE, node 1 of 3
process.roles=broker,controller
node.id=1
controller.quorum.voters=1@k0:9093,
2@k1:9093,3@k2:9093
listeners=PLAINTEXT://k0:9092,
CONTROLLER://k0:9093
advertised.listeners=PLAINTEXT://k0:9092
inter.broker.listener.name=PLAINTEXT

num.network.threads=8
num.io.threads=16
socket.send.buffer.bytes=1048576
socket.receive.buffer.bytes=1048576

log.dirs=/data1/kafka,/data2/kafka
num.partitions=12
num.recovery.threads.per.data.dir=4
default.replication.factor=3
min.insync.replicas=2
unclean.leader.election.enable=false
auto.create.topics.enable=false

log.retention.hours=72
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
num.replica.fetchers=4
replica.lag.time.max.ms=30000
replica.fetch.max.bytes=4194304

compression.type=producer
message.max.bytes=1048576
offsets.topic.replication.factor=3
transaction.state.log.replication.factor=3
transaction.state.log.min.isr=2

Every non-default value prevents a specific pitfall: unclean.leader.election.enable=false — forbid "far-behind replica" becoming leader; data-loss baseline; enabling = trading data for availability. auto.create.topics.enable=false — prevent typo topic names auto-creating 1-partition 1-replica crippled topics; once data lands, partition count immutable. min.insync.replicas=2 — with acks=all guarantees at least two copies before success. log.dirs two independent disks — not directories, physical disks; Kafka spreads partitions across disks, two disks = double IOPS. num.io.threads=16 — rule of thumb: 8-16× disk count, but < 2× CPU cores; oversizing causes contention.

One critical config not in file: JVM heap fixed at 6 GB, remaining 26 GB for page cache . Kafka officially recommends heap ≤ 6-8 GB — larger heap = longer GC pauses (many partitions → GC can take seconds, directly triggers rebalance), and heap steals memory from page cache. Use G1, set MaxGCPauseMillis=20, no huge heaps — biggest ops difference vs typical Java apps.

2.2 Config 2: Client (Idempotent + Manual Commit + Cooperative Rebalance)

# ---- producer ----
bootstrap.servers=k0:9092,k1:9092,k2:9092
acks=all
enable.idempotence=true
max.in.flight.requests.per.connection=5
retries=2147483647
delivery.timeout.ms=120000
request.timeout.ms=30000
linger.ms=10
batch.size=131072
buffer.memory=67108864
compression.type=lz4
max.request.size=1048576

# ---- consumer ----
bootstrap.servers=k0:9092,k1:9092,k2:9092
group.id=order-consumer-g1
enable.auto.commit=false
auto.offset.reset=earliest
max.poll.records=200
max.poll.interval.ms=600000
session.timeout.ms=45000
heartbeat.interval.ms=3000
fetch.min.bytes=1048576
fetch.max.wait.ms=500
partition.assignment.strategy=
CooperativeStickyAssignor
isolation.level=read_committed

Producer half = "reliable + high throughput" standard combo: enable.idempotence=true enables idempotence (broker deduplicates by PID+sequence; retries won't duplicate); only with idempotence can max.in.flight safely be 5 — without idempotence, in-flight > 1 + retry = reordering, a sequencing pitfall many miss; delivery.timeout.ms must exceed linger + request.timeout or client times out before batch even sends; lz4 compression best throughput/CPU balance (better ratio than snappy, faster than zstd).

Consumer half = "stable" standard combo, four critical settings: enable.auto.commit=false — auto-commit is #1 source of duplicates and loss; it commits by time, ignoring whether business processing finished; max.poll.records=200 — must match your per-batch processing time; Incident 1's lesson: if 200 records take 3 min, drop to 50 or make processing async; max.poll.interval.ms and session.timeout.ms must be separated — former = "processing too slow" tolerance (default 5 min), latter = "process dead" detection speed (45 s no heartbeat → kick); completely different semantics; CooperativeStickyAssignor — cooperative rebalance, only migrates necessary partitions, not eager's full-stop-then-reassign; rebalance blast radius from "whole group halts" to "individual partitions pause".

2.3 Partition Count: One Formula, Three Constraints

Partition count is hardest to change (only increase, and increasing breaks key ordering), so must get right once. Empirical formula: Partitions = max(target write throughput ÷ single-partition write throughput, target read throughput ÷ single-partition read throughput, target consumer concurrency) . Single-partition write throughput: HDD ~10-30 MB/s, SSD ~50-100 MB/s; single-partition read higher (page cache + zero-copy). Consumer concurrency is usually the tightest constraint : 24 consumer instances → at least 24 partitions, else extra instances idle.

Three hard constraints simultaneously: ① Single-broker partitions ≤ 2,000-4,000 (each partition has open file handles, indexes, replication threads; more partitions → slower controller recovery, full leader election can take tens of seconds); ② Cluster total partitions ≤ 200,000 (KRaft higher, but client metadata grows); ③ Partition count should be integer multiple of consumer instance count — else "some consumers get 3 partitions, some 2" skew; scaling amplifies skew. Anti-pattern: 200 partitions for a topic growing 5 GB/day → 25 MB/day per partition — wastes resources, tiny batches per consumer, terrible compression, lag monitoring becomes 200 lines.

2.4 Create Topic & Seven Pre-Launch Validations

kafka-topics.sh --bootstrap-server k0:9092 \
--create --topic order-event \
--partitions 24 \
--replication-factor 3 \
--config min.insync.replicas=2 \
--config retention.ms=604800000 \
--config compression.type=producer \
--config max.message.bytes=1048576

# Immediately verify replica distribution
kafka-topics.sh --describe --topic \
order-event --bootstrap-server k0:9092
# Watch three things:
# 1) Each partition's Leader evenly distributed
# 2) Replicas cross-rack
# 3) Isr equals Replicas

Seven validations, pre-launch checklist — miss one and it surfaces ugliest during failure: ① ISR == replica count — check immediately after creation; if not, investigate; ② End-to-end stress test — kafka-producer-perf-test at 1.5× target throughput, watch P99 and ISR jitter; ③ Kill one broker — verify election time (should be seconds), producer auto-retry succeeds, consumer duplicates appear; ④ Disk water-level alerts — 70% warn, 80% critical, 85% must act; don't wait for 100%; ⑤ Lag monitoring — per consumer group, threshold derived from business-acceptable latency; ⑥ Leader distribution balance — single broker leader count ≤ 1.3× average; ⑦ Offset commit success rate — rarely monitored, but first to drop when coordinator hiccups.

Core Source Code: Four Segments That Decide Behavior

3.1 Log.append: How Segmented Log Locates a Message

Kafka partition isn't one giant file; it's split into 1 GB segments , each with two sparse indexes: .index (offset → physical position) and .timeindex (timestamp → offset). Indexes are sparse — default every 4 KB ( index.interval.bytes=4096), so lookup flow: binary search index for "largest offset ≤ target", then sequential scan from that physical position. That's O(log n) + small sequential read, not O(n) scan .

// Log.scala  append()
val segment = maybeRoll(records, appendInfo)
// 1) Current active segment full 1 GB → roll new segment
segment.append(largestOffset, records)
// 2) Write FileRecords (into page cache)
if (unflushedMessages >= flushInterval)
flush(offset)
// 3) Update LEO, async flush
updateLogEndOffset(nextOffset)

// Read: findSegment -> binary search index -> sequential scan
val seg = logSegments.floor(offset)
val pos = seg.offsetIndex.lookup(offset)

This code explains two phenomena: ① Kafka doesn't fsync by default — message returns once in page cache; durability via OS flush + multi-replica redundancy; "Kafka faster than DB" essence: "it outsourced persistence to OS + replicas"; ② Lookup is two-phase — find segment then find position; many segments (tens of thousands per partition) slows lookup, so log.segment.bytes can't be too small or index files explode.

3.2 sendfile: Two Lines Carry Read Performance

// FileRecords.writeTo
public long writeTo(TransferableChannel ch,
long pos, int count) {
return ch.transferTo(this.start + pos,
count, this.channel);
}
// transferTo internally uses sendfile(2)
// If TLS enabled: falls back to read/write loop

This method's entire value: "data never enters user space". Operational corollary: read performance highly depends on page cache hit rate . Real-time consumers reading latest messages — data just written, still in page cache, read at memory speed; catch-up consumption (reading from 3 days ago) — page cache long evicted, all random disk reads , throughput drops an order of magnitude — why "backfill" tasks must be rate-limited; they saturate disk IO, impacting normal writes.

3.3 maybeShrinkIsr: When Does a Replica Fall Out of ISR?

ISR shrinkage has one criterion: follower's LEO lags leader's time beyond replica.lag.time.max.ms (default 30 s) . Note: it's time, not message count — old versions had "lag by N messages" dimension, removed because low-traffic topics caused false positives. Follower stays in ISR as long as it keeps sending fetch requests and catching up ; once stuck > 30 s, it's evicted.

// ReplicaManager.maybeShrinkIsr
val outOfSync = partition.isr.filter { r =
now - r.lastCaughtUpTimeMs maxLagMs
}
if (outOfSync.nonEmpty) {
partition.shrinkIsr(outOfSync)
// When ISR < min.insync.replicas
// acks=all writes throw NotEnoughReplicas
}

That last comment is the article's most critical conclusion: ISR shrinks below min.insync.replicas → acks=all writes fail immediately , producer gets NotEnoughReplicasException. This is protection, not bug — it refuses to treat "only one copy written" as success. So min.insync.replicas is an availability/consistency switch: set 2 (RF=3) = tolerate 1 broker failure while still writing; 2 failures = stop writes; set 1 = never stop for replica shortage, but equals abandoning redundancy. Conversely, Incident 3's problem wasn't misconfiguration — nobody monitored ISR — ISR dropping from 3 to 2 should have alerted instantly.

3.4 Offset Commit: One Commit = One Write

// GroupCoordinator.handleCommitOffsets
val records = offsets.map { case (tp, off) =
new SimpleRecord(
offsetKey(group, tp),
offsetValue(group, off))
}
// Batch write back to __consumer_offsets partition
appendForGroup(group, records)
// Failure returns COORDINATOR_LOAD_IN_PROGRESS
// Client retries, offset stuck at old value

This logic explains three production phenomena: ① Commit has latency — not local variable write; one network round-trip + one log append; tens of ms at peak normal; ② Commit can fail — coordinator migrating partition or just restarted returns LOAD_IN_PROGRESS, client retries; if process killed during retry, offset stays old, restart → duplicate consumption; ③ Commit is batched — client merges multiple partitions' offsets into one request; "commit after every record" is massive waste. Correct pattern: batch commit — finish batch, business transaction commits, then commit that batch's offsets — idempotence + transaction + offsets form "exactly-once" loop (covered in article 08).

Advanced Topics: Seven Accounts You Must Calculate Upfront

4.1 Capacity Account: Three Multiplied Then ×1.3

Disk capacity = daily write × retention days × replica factor × 1.3 . 1.3 covers three overheads: index files (~10%), segment tail not yet deleted, promo peaks. Example: 2 TB/day, 3 days, RF=3 → 2×3×3×1.3 ≈ 23.4 TB, three machines each 8 TB. Incident 2 missed the "peak" factor — sized for normal traffic, triple traffic exploded instantly. Second must-calc: network — write volume × RF = cross-machine traffic; 2 TB/day × 3 ≈ 70 MB/s, 10 GbE fine; but if mirror cluster (MirrorMaker), cross-DC traffic doubles — many dedicated lines saturated this way.

4.2 Ordering Account: Key Decides Partition, Partition Decides Order

Kafka only guarantees single-partition ordering ; cross-partition no guarantee. So "need ordering?" directly decides partition strategy: ① Global ordering — only 1 partition, throughput = single-machine limit, almost nobody does this; ② Per-business-key ordering — use order ID, user ID as key; same key → same partition; 99% of scenarios; ③ Fully unordered — key=null, round-robin or sticky partition, highest throughput. Most dangerous move: adding partitions to existing topic — hash(key) % partition_count modulus changes, same key lands new partition, previously ordered messages instantly scrambled. If must expand: new topic + dual-write + migrate, NOT direct partition increase.

4.3 Large Message Account: Four Gates Must All Change

Default max message 1 MB; to increase, must change all four simultaneously, miss one → RecordTooLargeException: ① broker message.max.bytes ; ② topic max.message.bytes ; ③ producer max.request.size ; ④ consumer fetch.max.bytes + broker replica.fetch.max.bytes (latter not changed → replica sync fails, ISR shrinks). But more important advice: don't stuff large messages into Kafka . 10 MB message kills batching, compression ratio tanks, page cache evicted by single message, drags all small messages in same partition. Correct pattern: message holds only object-storage reference (URL + metadata), consumer pulls on demand .

4.4 Reliability Account: acks + min.insync Matrix

These two parameters must be viewed together; discussing either alone is meaningless: acks=0 — fire-and-forget, may lose; acks=1 — wait leader only; leader crashes before sync → lose; acks=all + min.insync.replicas=1 — equals acks=1 reliability (ISR may be just leader); acks=all + min.insync.replicas=2 + RF=3 — standard "no-loss" config; cost: tolerate 1 broker failure, 2 failures = stop writes. Plus baseline: unclean.leader.election.enable=false — else far-behind replica may become leader, truncating data. Finance/orders use this; logs/tracking can relax to acks=1 for throughput.

4.5 Rebalance Account: Three Triggers, Two Costs

Rebalance triggers only three: ① Membership change (instance up/down/kicked); ② Subscribed topic or partition count change (adding partitions always triggers); ③ Coordinator change . Two cost types: Eager protocol = full group halt — all members release all partitions, re-join, re-assign, re-fetch offsets; tens of thousands partitions → tens of seconds; Cooperative protocol = partial pause — only migrates necessary partitions, others keep consuming. So 2.4+ versions always use CooperativeStickyAssignor. Another overlooked metric: rebalance count — >3/day → investigate; most common: max.poll.interval timeout (Incident 1) and GC pauses causing session timeout.

4.6 Isolation Account: Quotas Prevent One Business Dragging Whole Cluster

Multi-tenant cluster's nightmare: one business bursts, saturates bandwidth or disk. Kafka has two quota types: producer/consumer byte-rate quotas and request-rate quotas , settable per client.id or user; exceeding doesn't reject but briefly "throttles" the client — note this throttle time counts in client request latency, so monitoring shows "business RT spikes but broker load normal" — that's quota working. Other isolation: physical separation — core trading vs behavioral tracking separate clusters; cost higher but fault domains fully split. Decision simple: do both businesses' peaks coincide? — yes → split clusters.

4.7 Selection Account: Kafka, RocketMQ, Pulsar — How to Choose

Positioning differences matter more than parameter tables: Kafka — excels at throughput (million QPS), ecosystem (Flink/Spark/CDC full support), log-style scenarios; weak at many topics performance, no delayed messages or arbitrary timestamp replay (offset-only). RocketMQ — excels at business features (transaction messages, delayed messages, message trace, DLQ), handles many topics per machine; weaker throughput, smaller ecosystem. Pulsar — excels at storage-compute separation (stateless brokers, second-scale scaling, multi-tenancy + cross-region replication); weaker ops complexity, page-cache advantage diluted by BookKeeper. Pragmatic verdict: logs/stream processing/data pipelines → Kafka; transactional business messages → RocketMQ; multi-tenant large-scale elasticity → Pulsar .

Learning Path: From "Can Use" to "Dare to Tune"

Step 1: Spin up three-node single-machine cluster — KRaft mode, three brokers, create 3-partition 3-replica topic, kafka-topics --describe watch leader distribution, kill one, describe again, watch ISR change live.

Step 2: Break the two timeouts — make consumer exceed max.poll.interval, observe rebalance logs and lag curve; set session.timeout to 6 s, see if network jitter causes frequent kicks.

Step 3: Read three source-code sections — Log.append for segmentation & indexing, FileRecords.writeTo for zero-copy, maybeShrinkIsr for ISR judgment.

Step 4: Run one drill — calculate capacity per 4.1, run seven validations per 2.4, finally deliberately fill a disk, watch broker exit and downstream perception. Drill's value isn't success — it's discovering "thought configured but actually not effective" .

High-Frequency Interview Questions (with Answers)

Twenty Q&A pairs covering: why Kafka is fast (sequential write, zero-copy sendfile, batching+compression, OS page cache not JVM heap); partition/replica/ISR relationship; HW vs LEO; producer acks semantics; why idempotence required for in-flight=5; end-to-end no-loss config; duplicate consumption causes; consumer count > partition count; rebalance triggers & reduction; max.poll.interval.ms vs session.timeout.ms; offset storage & restart resume; replica count limits & costs; unclean leader election; partition sizing & mutability; zero-copy & TLS failure; retention strategies (delete + compact); Controller vs Coordinator; KRaft solutions; disk-full behavior & prevention; lag troubleshooting steps.

One-Page Cheat Sheet

Model : topic → partition (physical log, parallelism unit) → replica (copy, fault unit); single-partition ordered, consumers ≤ partitions. Write : batch + compress → leader append page cache (no fsync) → follower fetch sync → ISR all caught up → HW advances → acks=all responds. Read : sendfile zero-copy, data never touches JVM heap; read up to HW only; catch-up reads bypass page cache, must throttle. Reliability : acks=all + min.insync.replicas=2 + RF=3 + unclean=false + idempotence; all four mandatory. Consume : disable auto-commit, commit after batch success; max.poll.interval guards processing time, session.timeout guards liveness; use CooperativeStickyAssignor. Capacity : daily × retention × RF × 1.3; single broker 2000-4000 partitions; JVM heap 6-8 GB, rest for page cache. Two Must-Watch Metrics : ISR == replica count? Lag continuously rising?

Summary & Next Episode Preview

Kafka's complexity isn't in APIs but in relationships among three number groups : partition count sets parallelism, replica count sets fault tolerance, ISR sets real-time available replicas — any misconfiguration fails silently while "looking normal". Three lessons: Partition = parallelism unit = fault unit — all capacity & reliability trace to partition; ISR = reality value — replica count is design value; not monitoring ISR = having no replicas; Consumer group stability = two budgets — heartbeat budget + processing-time budget; exceed either → rebalance. Next episode: Producer internals — batching, compression, idempotence, transactions — dissecting every decision point of "sending a message". Includes complete high-reliability producer config and a "message duplicated 300 times" debug postmortem.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

KafkaPerformance Tuningzero-copydistributed messagingKRaftincident analysisbroker configurationconsumer groupsISRrebalance
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.