Kafka Replication: ISR, High Watermark, and the Hidden Dangers of acks=all
This article explains Kafka's replication mechanism through ISR and high watermark, revealing why acks=all waits only for in-sync replicas, how the 30-second shrink window works, why min.insync.replicas is essential, and the trade-offs of unclean leader election, with production configurations, source code insights, and incident case studies.
The previous four articles covered cluster formation, message ingestion, consumption, and storage. This article addresses a more fundamental question: why can Kafka claim messages are never lost? The answer lies in the replica mechanism — Kafka's entire reliability arsenal condensed into four concepts: LEO, HW, ISR, OSR. These terms seem simple, but the costliest production accidents — data loss despite acks=all, silent degradation when ISR shrinks to one, unclean election losing tens of seconds of orders — all stem from the combinatorial relationships among these concepts. This article dissects those relationships.
Three main threads: ① acks=all waits for replicas in ISR, not all replicas — when ISR shrinks it silently degrades, requiring min.insync.replicas as a safeguard; ② ISR is a dynamic list: evicted after 30 seconds without Fetch, reinstated only when LEO catches up — judgment occurs on the Fetch request path, no background thread; ③ Leader election prefers ISR members — the root of zero loss; the unclean switch is a last resort trading data for availability.
Incident 1: acks=all configured, orders still lost. Team verified replication.factor=3, acks=all, felt safe. One day broker-2 disk failed; during repair broker-1 restarted, partition ISR shrank to only the leader. At that point acks=all semantics silently degraded to "wait for leader local write"; writes succeeded, business unaware. Half hour later leader host lost power, last 11 seconds of order messages gone. acks=all without min.insync.replicas is a blank check. Incident 2: Follower kicked out of ISR, didn't return for three hours. A broker's aging disk spiked write latency to 800ms; follower fetches slowed, leader evicted it via the 30-second window. Disk later recovered, but the partition had 800GB history: follower had to replay hundreds of GB before rejoining. No replication throttling configured, catch-up traffic competed with production, producers timed out — ops restarted broker, catch-up reset, oscillation ensued. Getting kicked out isn't fatal; failing to return is. Incident 3: Switch opened for "availability" lost 47 seconds of orders. Ops tired of occasional partition unavailability enabled unclean.leader.election.enable. A network blip caused all three replicas to be briefly deemed unavailable; Controller elected a leader from out-of-sync replicas — it lagged the old leader by 47 seconds. Those 47 seconds of already-acknowledged order messages vanished permanently; when old leader recovered its log was truncated to align, leaving no chance for compensation.
1. Technical Analysis: Replica Mechanism's Entire Vocabulary — Just Four Terms
1.1 Replica Essence: A Replayable Log Plus Two Offsets
Each partition has replication.factor replicas on different brokers. A replica is not a "database slave"; it is a full copy of the same log : leader handles all reads/writes; followers only continuously send Fetch requests to leader, appending new messages to their own log ends. Two offsets describe this mechanism: LEO (Log End Offset, each replica's log end offset, i.e., "how far I've copied") and HW (High Watermark, i.e., "offset everyone agrees is fully replicated").
Three often-overlooked facts. First, HW = min(LEO of all ISR replicas) , a single global value maintained by leader and propagated to followers via Fetch responses; it is not a "majority offset" but "the slowest in the sync list". Second, consumers can only read up to HW — messages written on leader but not yet synced across ISR are invisible to consumers, guaranteeing they never read data that might later be lost in an election. Third, followers serve no external reads; all read traffic goes to leader — opposite of many storage systems' read/write separation. Reading from a follower would return stale data; Kafka simply doesn't offer that option.
1.2 ISR: A Dynamic "Caught-Up" List
ISR (In-Sync Replicas) = leader itself + all followers that are caught up; laggards move to OSR (Out-of-Sync Replicas). Key word: dynamic — this list is not static config but runtime state, changing with every Fetch request. Since 0.9 the sole criterion is: replica.lag.time.max.ms (default 30 seconds) — has a Fetch request arrived within that window? If yes, it's alive even if it fetched little data; if no Fetch for >30 seconds (slow disk, GC pause, network blip, process hang — all count) it's evicted. Early versions also used "lagging message count" (lag.max.messages), but high-throughput topics caused constant false evictions, so it was removed.
Two commonly misunderstood points. First, eviction logic runs on the leader's Fetch request handling path — no background thread "patrols"; if a follower stops sending Fetch, leader has no way to know its health. Second, every ISR change is written to the metadata layer (ZooKeeper in old versions, KRaft log in new), and Controller, admin commands, and monitoring panels all see this authoritative list.
1.3 What acks=all Actually Waits For: ISR, Not All Replicas
The precise semantics of acks=all (equals -1) is: wait until every replica in the current ISR has written the message . Note the three letters "ISR" — replicas not in ISR (OSR) do not participate in acknowledgment. This hides a trap: when ISR shrinks to only the leader, acks=all degrades to waiting for leader local write, equivalent to acks=1, reliability drops to zero, and the producer sees no error . That's the root cause of Incident 1.
The fix is min.insync.replicas: a broker-side (topic-level more common) configuration setting an ISR floor; when ISR count falls below the floor, acks=all writes fail immediately (throw NotEnoughReplicasException), trading "temporary unavailability" for "no silent downgrade". Hence the reliability trio must be deployed together: RF=3 provides redundancy, min.insync.replicas=2 draws the line, acks=all acknowledges against that line. Missing any one, the previous configs are pierced by the subsequent vulnerability.
1.4 Shrinkage: 30-Second Window, Eviction and Reinstatement Are Asymmetric Rules
Eviction and reinstatement criteria are asymmetric , a pitfall when troubleshooting "ISR won't come back". Eviction: 30 seconds without receiving a Fetch. Reinstatement: LEO must truly catch up to leader's current LEO — even one message short blocks reinstatement. This means a follower lagging 500GB, even after network recovery and machine health, must first copy 500GB before rejoining — during which it stays in OSR, absent from acks=all's acknowledgment list.
Catch-up traffic must also be managed: follower replay traffic can far exceed normal production traffic, saturating production bandwidth. Kafka offers replication quotas (replication quota) to set byte quotas for both follower and leader. Throttling is a double-edged sword — too low slows catch-up, delaying ISR recovery; too high equals no limit. In production, set a "maximum that doesn't impact production P99 latency" and proactively raise it before known catch-up storms like scaling or broker replacement.
1.5 HW and Consumer Visibility: Why HW Always Lags One Beat
Write acknowledgment (acks=all returns) and consumer visibility (HW advance) are two misaligned moments : when all ISR replicas have written, ack returns immediately; but HW must wait for the follower's next Fetch round-trip for leader to update and propagate — so HW naturally lags "already acknowledged" by one Fetch cycle (normally a few ms, under pressure hundreds of ms). This yields a troubleshooting heuristic: HW stuck on monitoring doesn't mean writes are stuck, but writes stuck always precede HW stuck . Conversely, when a follower crashes, HW freezes at old position — consumers can't read new data while producers still get normal acks — this "write succeeded but can't read" brief window is normal, not message loss.
1.6 Leader Election: Picking from ISR Is the Root of Zero Loss
After leader crashes, Controller uses the ISR list from metadata to elect: if ISR has any live replica, pick one from ISR . Why does picking from ISR guarantee zero loss? Because HW is defined as "offset all ISR replicas have written" — any ISR replica possesses all messages up to HW; whoever becomes leader won't lose acknowledged data. Note election happens on Controller side; Controller's ISR view may lag leader's final knowledge by one beat (metadata propagation delay), occasionally causing a replica leader considered evicted to be elected — a source of transient jitter but not breaking HW semantics.
When ISR is entirely gone (e.g., whole rack power loss), Controller faces a choice: default unclean.leader.election.enable=false keeps partition offline, waiting for ISR replicas to return — trading availability for consistency ; enabling the switch picks a leader from OSR, partition becomes available immediately, but new leader lacks the lagging segment, loss is permanent (old leader's log truncated on recovery). This switch is an architectural trade-off, not something an ops person should flip to silence alerts: it effectively shifts the cluster from CP to AP, while Kafka's message semantics, upstream retry design, and downstream idempotent handling are all designed for CP.
1.7 Not Coming Back: A State More Alarming Than Eviction
Incident 2's lesson deserves its own section: eviction takes seconds, reinstatement takes hours . A large partition's follower catch-up speed is bounded by three factors — local disk read capacity, leader-side network and read pressure, replication quota. Assume 1TB backlog, catch-up at 200MB/s → 85 minutes to catch up; during that time this replica provides zero reliability redundancy. So an ISR eviction alert's real meaning isn't "one less replica" but "for the next N hours this partition runs in degraded mode" — if combined with min.insync.replicas=2 and RF=3, losing one more replica makes the partition unwritable. Ops must assess catch-up duration immediately upon eviction, not wait for self-healing.
2. Production Practice: Two Complete Configurations and an ISR Postmortem
2.1 Config 1: Broker-Side Full Replica Configuration
This continues the server.properties replica section from article 01, each annotated with the pitfall it prevents, ready for production.
# ===== Replica & ISR: broker side =====
# Default replication factor for new topics (still recommend explicit per topic)
default.replication.factor=3
# ISR floor: when below this, acks=all writes fail directly
# This is the safety rope for acks=all, must not be omitted
min.insync.replicas=2
# Shrink window: evict from ISR if no Fetch received for 30s
# Stops bleeding on slow-disk machines; don't increase just to reduce alerts
replica.lag.time.max.ms=30000
# Prohibit OSR promotion: prefer partition unavailability over data loss
unclean.leader.election.enable=false
# Follower fetch: max bytes per fetch
replica.fetch.max.bytes=1048576
replica.fetch.response.max.bytes=10485760
# Fetch threads: increase for brokers with many partitions
num.replica.fetchers=4
# Replication throttling (bytes/sec): default unlimited
# Keep off normally; adjust dynamically during scaling / broker replacement
# leader.replication.throttled.rate=52428800
# follower.replication.throttled.rate=52428800
# Auto balance: migrate leader back when skew > 10%
auto.leader.rebalance.enable=true
leader.imbalance.per.broker.percentage=10
# Max 500 partitions per rebalance to avoid migration storm
leader.imbalance.check.interval.seconds=3002.2 Config 2: Topic-Level Trio and ISR Health-Check Script
Topic-level config always overrides broker-level — the lesson from Incident 2 about explicit params at topic creation applies equally to replica params. Standard topic creation: deploy the trio together, plus a health-check script for crontab:
# Create topic: trio deployed at once
BS=localhost:9092
UC=unclean.leader.election.enable=false
kafka-topics.sh --bootstrap-server $BS \
--create --topic order-event \
--partitions 24 --replication-factor 3 \
--config min.insync.replicas=2 --config $UC
# ===== ISR health-check script (crontab every 5 min) =====
#!/bin/bash
BS=localhost:9092
# 1. Partitions below min.isr: writes are being rejected
UR=$(kafka-topics.sh --bootstrap-server $BS \
--describe --under-minisr-partitions)
if [ -n "$(echo "$UR" | grep Topic)" ]; then
echo "P0: UnderMinIsr!" ; echo "$UR"
fi
# 2. Partitions with incomplete ISR: degraded, watch duration
kafka-topics.sh --bootstrap-server $BS \
--describe --under-replicated-partitions \
| awk '{print $2, $6}' \
| sort | uniq -c
# 3. Partitions with ISR size long-term < RF need manual intervention:
# check if "catching up" or "can't come back"The script's layered logic is deliberate: UnderMinIsr is P0 (writes failing, minute-level response); UnderReplicated is P1 (degraded, watch duration — self-healing within 10 minutes is noise, persistent timeout needs human). A single generic "ISR not full" alert either misses the fatal or drowns in noise.
2.3 Postmortem: 12 Minutes from ISR 3 to 1
Dissect a real ISR event on a timeline (matches Incident 1's cluster, min.insync.replicas was still 1 at the time):
T+0s : broker-2 GC pause 40s, leader evicts it at 30s, ISR 3→2.
T+42s : broker-1 in same datacenter evicted due to switch port flapping, ISR=1. acks=all degrades to local write, no alert (UnderReplicated alert threshold "5 consecutive" still waiting).
T+6min : broker-1 network recovers, starts catching up 3 minutes backlog, T+9min reinstated, ISR=2.
T+12min : broker-2 catches up 11 minutes backlog, ISR=3. Entire 12 minutes all writes existed only on leader single replica — luckily leader didn't crash, unfortunately no one knew this 12-minute risk window existed.
Remediation actions are exactly this article's configs: min.insync.replicas=2 (make writes fail from T+42s, triggering client retry instead of silent bare-metal run), UnderMinIsr P0 alert (alarm at ISR=2, not after disaster), GC pause included in ISR eviction root-cause (40s pause is JVM tuning issue, not Kafka's fault). ISR events are essentially distributed system "health-check windows" — they tell you the current single-point risk exposure, not wait for actual loss to postmortem.
2.4 Rack Awareness: Scatter Replicas Across Fault Domains
Replica mechanism guards against single-machine failure; rack awareness guards against rack/datacenter-level failure : configure broker.rack on brokers, Controller automatically spreads replicas of same partition across different racks during assignment. Note it only guarantees initial distribution — broker scaling, partition reassignment, leader migration won't auto-correct skew; periodic checks needed. Also, if two "racks" are actually two racks in same datacenter, it only survives rack power loss, not datacenter network partition — true disaster recovery requires cross-datacenter replicas, but cross-DC latency directly increases acks=all acknowledgment latency and ISR false-eviction rate, so replica.lag.time.max.ms must be widened accordingly .
3. Core Source Code: Four Key Locations
From 2.8/3.x partition and state machine source, keeping main logic, hiding engineering details. Four snippets form a complete lifecycle from "write" to "ack" to "crash takeover".
3.1 Partition#maybeShrinkIsr: Eviction Judgment
Shrink triggered by leader in periodic task (translating "30 seconds no Fetch" to code: "last active time exceeds maxLagMs"):
def maybeShrinkIsr(): Unit = {
// Candidates: ISR followers (leader exempt)
val candidates = inSyncReplicas.filterNot(_.isLeader)
// Those without Fetch in 30s are lagging
val lagging = getOutOfSyncReplicas(candidates, maxLagMs)
if (lagging.nonEmpty) {
val keep = candidates.filterNot(lagging.contains)
// List change written to metadata layer, cluster-wide visible
updateIsr(keep.map(_.brokerId))
info(s"Shrunk ISR ... lagging: $lagging")
}
}Note "last active time" refresh point is on Fetch request handling path — as long as follower sends Fetch, even fetching zero messages, it's considered active. That's why a "half-dead" follower only returning empty responses isn't evicted, while a healthy follower with 31s GC pause gets false-evicted.
3.2 Partition#updateReplicaLogInfo: Reinstatement Judgment
Expansion occurs when leader processes follower Fetch: updates that follower's LEO and checks catch-up:
// Every follower Fetch arrives here
def updateReplicaLogInfo(followerLEO: Long): Unit = {
// Step 1: record reported LEO
replica.state.logEndOffset = followerLEO
// Step 2: only reinstate when caught up to leader's current LEO
if (followerLEO >= localLog.endOffset)
expandIsr(replica.brokerId)
}Reinstatement condition is "greater than or equal to leader's current LEO", not "lag below threshold". This guarantees that at reinstatement moment the replica owns all data up to HW, eligible for election and acks=all acknowledgment — every member on the list is always ready to take over , the core value of ISR semantics.
3.3 Partition#checkEnoughReplicasReachOffset: acks=all Wait Logic
Produce request hangs on DelayedProduce; every ISR change re-checks completion:
def checkEnoughReplicasReachOffset(
requiredAcks: Int,
requiredOffset: Long): Try[Boolean] = {
// Count ISR replicas that have written to requiredOffset
val numAcks = inSyncReplicas.count(_.state.logEndOffset >= requiredOffset)
val minIsr = config.minInSyncReplicas
// ISR shrunk below floor: fail fast, don't wait
if (numAcks < minIsr)
return Failed(new NotEnoughReplicasException)
if (requiredAcks == -1) // acks=all
Success(numAcks >= inSyncReplicas.size)
else Success(true) // acks=1 only waits leader
}Two branches worth chewing: minIsr not met → "fail" not "keep waiting" — waiting only wastes producer timeout, better let client retry immediately; acks=all success condition counts inSyncReplicas.size — current ISR full membership — smaller ISR makes acknowledgment easier but less reliable, the exact hole min.insync.replicas plugs.
3.4 PartitionStateMachine#doElectLeader: Election Branches
private def doElectLeaderForPartitions(...) {
// Controller perspective: any live replicas in ISR?
val liveIsr = isr.filter(isLive(_))
if (liveIsr.nonEmpty) {
// Default path: elect from ISR. Any elected replica
// owns all data up to HW → zero loss
electLeaderFromIsr(...)
} else if (uncleanElectionEnabled) {
// OSR promotion: availability for consistency,
// loss = old leader LEO - elected replica LEO
electLeaderUnclean(...)
} else {
// Stay offline: wait for ISR replica to return
}
}Controller uses the ISR from metadata , not a broker's in-memory copy — explains the "lag one beat" in 1.6: leader just evicted, metadata not yet propagated, Controller may still think it's in ISR. This design picks "metadata as source of truth" between availability and consistency, gaining a cluster-wide single ISR view.
4. Advanced: Seven Accounts
4.1 Reliability Account: Trio Is a Chain
RF=3, min.insync.replicas=2, acks=all form a chain: RF provides redundancy raw material, min ISR defines "below how many is dangerous", acks=all acknowledges against ISR. Any link missing: RF=2 with min ISR=2 means one failure makes partition unwritable (availability too poor); min ISR=1 makes acks=all silently degrade (Incident 1); acks=1 makes all replica configs moot (write doesn't wait any follower). Standard answer is exactly one combo: 3/2/all, don't invent new combos.
4.2 Availability Account: Cost of min ISR Floor
min.insync.replicas=2 + RF=3 availability cost: any two replicas simultaneously down → partition unwritable. Unwritable window = probability of two simultaneous failures × recovery time. Normal ops probability negligible; but rolling restarts, datacenter maintenance — planned operations — drive probability to 100% — so SOP must specify: before maintenance confirm target partition ISR full, move one broker at a time, wait for ISR full recovery before next.
4.3 Bandwidth Account: Replica Traffic Is 2x Production
Steady state replica sync traffic = production traffic × (RF − 1). Production 200MB/s, RF=3 → inter-broker replication 400MB/s, size NICs and switches accordingly. Non-steady state worse: follower catch-up single-partition replication can hit disk limit; during scaling storms, provision NICs at "production peak × RF" and set replication throttle to "remaining production bandwidth".
4.4 Election Account: Unclean Loss Is Estimable
Unclean election loss ≈ old leader LEO − elected OSR replica LEO (times avg message size). Not abstract risk, but concrete bytes estimable via lag metrics. This yields an ops red line: Prerequisite for enabling unclean: business explicitly accepts "max lose X seconds of data" at that magnitude and downstream has reconciliation capability. Logs, metrics acceptable; orders, payments not.
4.5 Rack Account: Awareness of Distribution, Not Health
Rack awareness only handles "where to place new replicas", not "is rack still alive" — after rack B dies, Controller piles missing replicas onto rack A; when rack B recovers, distribution is skewed, needs manual rebalance trigger. Rack awareness must pair with periodic distribution checks , else six months later it's just "once cross-rack".
4.6 Monitoring Account: Five Metrics Decide Life/Death
UnderReplicatedPartitions: degraded partition count, >0 for 10 minutes → alert.
UnderMinIsrPartitions: P0, writes failing.
Per-partition ISR count / RF: trend matters more than threshold; long-term 2/3 means a replica can't return.
replica.lag time distribution: slow-disk brokers spike in "eviction count".
Leader ISR shrink/expand per minute: frequent jitter means 30s window too tight for this load; check disk and GC first, don't just widen window.
4.7 Version Account: Three Replica-Related Changes
Since 0.9, "lagging message count" judgment (replica.lag.max.messages) removed, only time window remains — because count-based eviction always false-kills on high-throughput topics.
Since 1.1, reassignment supports replication throttling (KIP-73), scaling no longer crushes production.
3.x KRaft moves ISR list from ZooKeeper into KRaft metadata log, change propagation faster, Controller election and metadata "lag one beat" window significantly shrinks, but semantics unchanged.
5. High-Frequency Interview Questions (with Answers)
Q1 · What is the exact semantics of acks=all? Why might it be unsafe?
Answer: Waits for all replicas in the ISR to write the message. Unsafe because ISR is a runtime list: when it shrinks to only the leader, acks=all degrades to waiting for local write, equivalent to acks=1 with no error. Hence min.insync.replicas is mandatory — writes fail fast when ISR falls below the floor, trading unavailability for no silent downgrade.
Q2 · What are LEO and HW? Who maintains HW?
Answer: LEO is each replica's own log end offset (how far I've copied); HW = high watermark = min(LEO of all ISR replicas), a single global value maintained by leader, propagated to followers via Fetch responses. Consumers only read up to HW, guaranteeing read data won't be lost after any ISR replica takes over.
Q3 · How do ISR eviction and reinstatement conditions differ?
Answer: Asymmetric. Eviction: no Fetch within replica.lag.time.max.ms (default 30s). Reinstatement: LEO must actually catch up to leader's current LEO, even one message short blocks it. So eviction is seconds, reinstatement can be hours (must replay backlog first); catch-up replication throttling must be considered separately.
Q4 · With min.insync.replicas=2 and RF=3, how many brokers can fail while still writable?
Answer: One. With one down ISR=2 equals floor, writes continue; second down makes ISR=1 below floor, acks=all fails. This config balances availability and reliability: to survive two failures need RF=4 + min ISR=3, costing more storage and bandwidth.
Q5 · Why does Kafka define HW as min(ISR LEO) instead of quorum?
Answer: Kafka's reliability model isn't quorum: it guarantees "data before HW never lost", and HW takes the slowest in the sync list, ensuring any ISR replica promoted has all acknowledged data. Combined with acks=all + min ISR, effect equals "at least N copies on disk before ack"; quorum models (Raft) require majority vote per write, different latency model but same safety goal.
Q6 · Why can consumers only read up to HW?
Answer: Data before HW exists on all ISR replicas; whoever becomes leader later won't lose it. Data after HW exists only on a subset; leader crash may cause new leader to truncate it. Allowing consumption beyond HW would let consumers read data that subsequently disappears, breaking "read implies acknowledged" semantics.
Q7 · Do followers serve reads? Why this design?
Answer: No, all reads and writes go to leader. Followers lag behind HW; reading them returns stale data, breaking ordering semantics. Kafka's read path relies on page cache; leader's hot data is already cached, reading followers doesn't save resources. To scale reads, increase partitions for parallelism, not read replicas.
Q8 · After a follower is evicted from ISR, what condition lets it return? What is it before return?
Answer: Condition: LEO catches up to leader's current LEO, judged by leader when processing its Fetch and expanded back. Before return it sits in OSR: doesn't participate in acks=all, no election eligibility, provides no redundancy. Large partition catch-up can take very long; during that time losing another replica may trigger UnderMinIsr, so monitor catch-up progress, don't just wait.
Q9 · What happens if unclean.leader.election.enable is turned on? When should it be enabled?
Answer: When ISR all dead, allows election from OSR (out-of-sync replicas), partition becomes available but loses data = old leader LEO − elected replica LEO, and old leader's log truncated on recovery — loss permanent. Only suitable for data-loss-tolerant, availability-first scenarios (logs, metrics); strictly forbidden for orders/payments. Before enabling, estimate "max seconds of data loss" via lag metrics and get business sign-off.
Q10 · How is a new leader elected after leader crash? Who elects?
Answer: Controller uses ISR list from metadata: picks a live replica from ISR (usually first in list). Election completes on Controller side, then broadcast cluster-wide. Note Controller's ISR may lag leader's final view by one beat (metadata propagation delay); extreme case may elect a replica leader already considered evicted, but HW semantics remain intact.
Q11 · Why was lag-by-message-count judgment removed after 0.9?
Answer: replica.lag.max.messages evicted based on lagging message count, but high-throughput topics have large steady-state lag (fetch batch in-flight) causing constant false evictions; low-throughput topics barely trigger it, no protection. Time window (30s no Fetch) is traffic-agnostic, semantics more direct: "are you alive?" not "did you copy?".
Q12 · Is replication.factor=2 acceptable?
Answer: Not recommended. RF=2 with min ISR=1 → any failure makes acks=all silently degrade to single-replica bare metal; with min ISR=2 → any failure makes partition immediately unwritable. Both ends blocked: either unreliable or too fragile. Production standard is 3/2/all. RF=2 only for extremely disk-cost-sensitive log archival clusters.
Q13 · Will a 40s GC pause cause ISR eviction? How to avoid false eviction?
Answer: Yes: no Fetch within 30s window → eviction. Correct fix is not widening window but eliminating 40s pauses: switch to ZGC/G1 tuning, limit heap object sizes, don't let broker process carry excessive heap. Widening window is "delaying stop-loss" — real disk failure you wait N more seconds to evict, acks=all acknowledgment list carries a dead replica longer.
Q14 · For cross-datacenter deployment, should replica.lag.time.max.ms be adjusted?
Answer: Must widen. Cross-DC RTT typically 1-5ms baseline, jitter up to tens of ms; Fetch round-trip slower. If inter-DC bandwidth limited, catch-up also slower. Set window to "10x cross-DC RTT + margin", commonly 60-120s. Meanwhile acks=all acknowledgment latency includes slowest DC round-trip; producer request.timeout.ms must also be widened accordingly.
Q15 · What do frequent ISR shrinks and long-term incomplete ISR indicate respectively?
Answer: Frequent shrinks: 30s window too tight for this load; root cause usually slow disk (write latency spikes), GC pauses, network jitter — fix root cause first, don't widen window to mask. Long-term incomplete (e.g., stuck at 2/3): a follower can't catch up and stay caught up; typical cause partition too large (catch-up exceeds 30s then re-evicted, death spiral) or replication throttle too tight — assess partition data volume vs catch-up bandwidth match.
Q16 · In kafka-topics.sh --describe, what do Isr and Leader mean? Troubleshooting order?
Answer: Leader is brokerId of current read/write replica; Isr is sync replica list. Troubleshooting order: first check for Leader=-1 or missing leader row (partition offline, P0); then check if Isr count < RF (degraded); then check if Isr contains already-decommissioned brokerId (metadata lag); finally for degraded partitions examine lag to judge catching up vs can't return.
Q17 · Why is "ISR event is essentially a health-check window"?
Answer: Because the moment eviction happens, system tells you: this partition's reliability redundancy just dropped to N-1 copies, and for the next few hours (catch-up period) it stays degraded. This info appears before actual data loss — golden time to fix: check disk, GC, add capacity, assess UnderMinIsr risk. Waiting until loss occurs is postmortem.
Q18 · What if min.insync.replicas > replication.factor?
Answer: Permanent write failure: ISR can never reach floor (ISR max = RF), all acks=all writes throw NotEnoughReplicasException. This is config error, not tuning issue; occurs when dynamically reducing RF but forgetting to lower min ISR, typical during scale-down or cost-cutting. Before changing RF, verify all topics' min ISR.
Q19 · Can acks=all producer retries cause duplicates? Related to replica mechanism?
Answer: Yes: acks=all response times out but write actually succeeded, retry produces duplicate. Related to replica mechanism — ISR fluctuations lengthen acknowledgment latency, increasing timeout retries. Fix: enable producer idempotence (enable.idempotence=true, PID+sequence dedup) or transactions, not disable retries. Replica mechanism guarantees "no loss"; idempotence guarantees "no duplicates"; two layers each handle their own.
Q20 · You're asked to reliability-review an e-commerce order topic. What do you check?
Answer: Checklist: ① Trio RF=3 / min ISR=2 / acks=all all present (per-topic check, prevent topic-level override); ② unclean switch false (both broker and topic level); ③ Replica distribution cross-rack (rack awareness effective, distribution not skewed); ④ ISR monitoring dual alerts configured (UnderMinIsr P0 + UnderReplicated P1); ⑤ ISR eviction drill done? Maintenance SOP includes "one broker at a time, wait ISR recovery"; ⑥ Producer enable.idempotence on, timeout/retry params match cross-DC latency.
6. One-Page Cheat Sheet
Four terms: LEO = how far I copied; HW = offset all agree copied (= min ISR LEO); ISR = caught-up list; OSR = laggard list. Consumers only read before HW.
Two rules: Eviction looks at "30 seconds no Fetch" (time), reinstatement looks at "LEO caught up" (data) — eviction fast, reinstatement slow; judgment on Fetch path, no patrol thread.
Trio: RF=3 gives redundancy, min.insync.replicas=2 draws floor, acks=all acknowledges against floor — missing one link, others pierced.
Two alerts: UnderMinIsr is P0 (writes failing); UnderReplicated sustained 10 minutes is P1 (degraded, first compute catch-up time). ISR event is health-check window, not accident.
7. Summary & Next Episode Preview
Replica mechanism revolves entirely around one list: ISR . It's the "caught-up" commitment list — ack=all acknowledges against it, election picks from it, consumers read what it collectively owns; its entry/exit rules asymmetric (eviction by time, reinstatement by data), its shrinkage causes silent reliability degradation, hence min.insync.replicas turns degradation into explicit failure. Remember three incidents' lessons: acks=all is a blank check (needs min ISR to cash), evicted replicas may never return (must compute catch-up time), unclean switch trades data for availability (must estimate cost via lag). Next article covers the command system behind this list: Controller & KRaft — who maintains cluster-wide ISR view? How much faster are elections and failovers after metadata moved from ZooKeeper to KRaft log? We'll start with a "Controller switch triggering chain re-elections".
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
