Skip to content
Kafka Controller & KRaft: Why Metadata Migration Slashes Failover from 5 Minutes to 4 Seconds

Kafka Controller & KRaft: Why Metadata Migration Slashes Failover from 5 Minutes to 4 Seconds

This article analyzes Kafka's Controller architecture evolution from ZooKeeper to KRaft, detailing how metadata migration to an internal Raft-replicated log transforms failover from minutes-long rebuilds to seconds-long handoffs, covering deployment models, migration pitfalls, source-code mechanisms, and production monitoring strategies.

Architecture Pony
Architecture Pony
Architecture Pony

Introduction: The Cluster's Command System

The previous article on ISR mentioned that ISR membership changes are written to the metadata layer. This article examines that metadata layer itself — the cluster's command system. Leader failures require reassignment, topics need creation, partitions need allocation, broker up/down events need detection — all these "cluster-level" operations fall to the Controller. In Kafka 3.x, the Controller architecture underwent a fundamental overhaul: metadata migrated from ZooKeeper into Kafka's own KRaft log. This is not merely a storage swap; it moves "the command system's information from someone else's database back into its own log" — changing failover speed, metadata scale limits, and operational mental models.

Three Main Threads

① The Controller is not a standalone service but an elected broker (in KRaft it can run as a separate process); failover speed depends on metadata rebuild time.

② KRaft's core step: metadata itself becomes a log (@metadata partition): append-only, idempotent replay, snapshots for acceleration; the Controller state machine is merely a materialized view of this log.

③ ZK-to-KRaft migration has a clear one-way door: migration mode allows rollback; finalize makes it irreversible.

Incident 1: ZooKeeper Jitters Once, Cluster Paralyzed for Ten Minutes

Network blips caused 40 brokers' ZK sessions to batch-timeout and reconnect. The Controller was forced to handle 40 deregistrations + 40 registrations + thousands of partition re-elections. Simultaneously, ZK-side watch event floods delayed the new Controller's metadata rebuild to minutes. During this, LeaderAndIsr broadcast RPCs saturated brokers, and normal business writes timed out. Metadata-layer jitter was amplified into a data-layer avalanche.

Incident 2: Batch-Creating 3,000 Partitions, Stuck Halfway

Business created 500 topics with 3,000 partitions total. The old Controller used a synchronous model: write to ZK one by one, wait for ack each time. The request queue grew; frontend timeouts triggered retries, doubling the queue. ZK write latency spiked; the Controller appeared hung; manual restart recovered it. Root cause: Controller single-point both computes and waits for I/O; synchronous RPC model cannot withstand metadata operation floods.

Incident 3: Migration Halfway, Dual-Write Config Clash

During ZK-to-KRaft migration mode, someone conveniently changed a topic config in ZK's kafka-configs — KRaft side didn't sync. In dual-write phase ZK is the authoritative source, but the KRaft log lacked that record. Later forgotten, after migration completed the config silently rolled back, throttling failed, triggering producer throttling at peak. The "two entryways" during dual-write are an accident breeding ground.

I. Technical Analysis: Controller Is the Commander, KRaft Is the New Command System

1.1 Who Is the Controller: A Broker Wearing a Hat, Not an Independent Kingdom

In the ZK era, every broker on startup raced to register the ephemeral /controller node in ZK; first come, first served — the winner moonlights as Controller, while others stay in a standby "I can take over immediately" state. Thus the Controller has built-in HA: if it dies, the ephemeral node disappears, remaining brokers race again. In the KRaft era this model changes: the Controller can be a separate process (dedicated mode) or co-located with the broker (combined mode), but leader election shifts from "racing for ZK ephemeral node" to Raft election inside the KRaft quorum , still 3 or 5 nodes yielding 1 active.

Figure 1: ZooKeeper era — metadata external, Controller is a broker wearing a hat, watch-driven everything
Figure 1: ZooKeeper era — metadata external, Controller is a broker wearing a hat, watch-driven everything

Figure 1 · ZooKeeper era: metadata external, Controller is a broker wearing a hat, watch-driven everything

1.2 Controller's Responsibilities: Five Things

① Partition leader election — after broker crash or ISR change, pick new leader from ISR (covered in article 05); ② Topic and partition management — create/delete, partition assignment, partition expansion; ③ Replica reassignment — executor for scale-out migration and rack rebalancing; ④ Broker lifecycle — up/down detection, quota distribution; ⑤ Config broadcast — dynamic configs (e.g., replication throttle from article 05) pushed to every broker. Common thread: all are metadata write operations . The Controller is essentially a "metadata change acceptance and execution engine"; its performance ceiling and failure recovery speed depend entirely on how metadata is stored.

1.3 ZK Era's Three Original Sins: Dual Views, Slow Rebuild, Broadcast RPC

First, dual views : metadata authority lives in ZK, but the Controller works off an in-memory view; they sync via watches — asynchronous, so windows of inconsistency inevitably exist (amplification effect in Incident 1). Second, slow rebuild : on Controller failover, the new Controller must read all metadata from ZK node-by-node to rebuild the in-memory view; million-partition clusters take minutes, during which the cluster has no commander. Third, broadcast RPC : state changes require the Controller to actively send LeaderAndIsr/UpdateMetadata requests to all brokers; more brokers and larger changes mean bigger broadcast floods — a synchronous blocking model where one large change can stall the Controller.

1.4 KRaft's Masterstroke: Metadata Itself Becomes a Log

Figure 2: KRaft — Controller quorum maintains @metadata log, brokers pull increments, watch era ends
Figure 2: KRaft — Controller quorum maintains @metadata log, brokers pull increments, watch era ends

Figure 2 · KRaft: Controller quorum maintains @metadata log, brokers pull increments, watch era ends

KRaft moves metadata from external ZK back into Kafka itself, using Kafka's strongest primitive: an append-only log (the @metadata partition). Every metadata change — create topic, elect leader, change config — becomes a record appended to this log, replicated by the Controller quorum (3 or 5 nodes) via Raft, committed only after majority ack. The Controller's in-memory state machine is merely a "materialized view" of this log : any Controller restart replays the log to reconstruct identical state.

The design's cleverness lies in reuse: the log's idempotent replay makes standby Controllers naturally hot standbys (everyone replays the same log, state always consistent); the log's snapshots avoid replay from zero (Figure 4 details); brokers and clients consuming metadata also become pulling log increments — using the same Fetch mechanism as regular topics; watches and broadcast RPCs are entirely retired.

Figure 4: Metadata log — segments before snapshot point cleaned, restart uses snapshot + incremental replay, log size bounded
Figure 4: Metadata log — segments before snapshot point cleaned, restart uses snapshot + incremental replay, log size bounded

Figure 4 · Metadata log: segments before snapshot point cleaned, restart uses snapshot + incremental replay, log size bounded

1.5 Event Queue: Controller's Single-Threaded Heart

The KRaft Controller is a strict single-threaded event queue model: all operations — write requests, Raft callbacks, session timeouts — are wrapped as events queued sequentially; the active Controller processes them on one thread. No lock contention (only one thread), no concurrency bugs (state changes serially) — code correctness dramatically improves. This directly addresses the ZK-era Controller's most criticized aspect: hundreds of concurrent paths mutating the same in-memory view, historically contributing countless hard-to-debug issues. The trade-off: single-thread throughput has a ceiling, so every KRaft Controller design choice supports it: slow operations (I/O) delegated to Raft replication and log append; fast operations (in-memory state machine changes) stay in the event loop . Batch-creating 3,000 partitions becomes batched log records, no longer synchronous waiting.

1.6 Broker Perspective: From "Passive RPC Receiver" to "Active Log Puller"

KRaft brokers run a MetadataListener : via IncrementalFetch long-polling they pull metadata increments from the active Controller (only records after their own lastOffset), apply to local MetadataCache, then drive log manager, replica manager, etc. For the broker, "metadata changed" is no longer a passive RPC arrival event but the natural result of consuming a message stream — uniform latency, guaranteed ordering, lost data can be re-fetched . This is why KRaft clusters scaling to hundreds of thousands of brokers is plausible: metadata distribution shifts from "fan-out broadcast" to "everyone pulls for themselves".

1.7 Failover Timing: Rebuild Becomes Handoff

Figure 3: Controller failover latency comparison — ZK era is 'rebuild', KRaft era is 'take over'
Figure 3: Controller failover latency comparison — ZK era is 'rebuild', KRaft era is 'take over'

Figure 3 · Controller failover latency comparison: ZK era is "rebuild", KRaft era is "take over"

Breaking both eras' failover into the same table makes the difference obvious. Detection phase is dominated by session timeout for both (seconds, deliberately designed — too fast would mistake network jitter for crash). The real gap is in takeover : ZK era must "pull state from someone else's database and rebuild"; KRaft era standbys have been continuously replaying the same log, state already in place, takeover only needs to complete Raft election and replay the log tail (milliseconds to seconds). Production-worthy conclusion: Controller failover lower bound is session timeout; upper bound depends on metadata rebuild — KRaft compresses the upper bound to the same order of magnitude as the lower bound.

II. Production Practice: KRaft Deployment Config & Migration Path

2.1 Config 1: Three-Node Combined Mode KRaft Cluster

Small-to-medium clusters (partitions under millions, light metadata pressure) recommended combined mode: each node is both broker and controller, saving machines; large clusters or heavy Controller load (frequent leader elections, large batch partition ops) use dedicated mode, controllers as separate 3 or 5 nodes. Complete three-node combined mode config:

# server.properties (node 1, combined mode)
# Node role: broker + controller dual role
process.roles=broker,controller
# Node ID: unique ID in quorum
node.id=1

# Controller quorum full roster:
# nodeID@host:port, all nodes must list all
[email protected]:9093,
[email protected]:9093,[email protected]:9093
# quorum minimum three (Raft majority write), five more stable

# Two listeners: separate business traffic from metadata traffic
listeners=BROKER://:9092,CONTROLLER://:9093
controller.listener.names=CONTROLLER
advertised.listeners=BROKER://10.0.0.1:9092

# Metadata log directory: recommend separate disk or directory
metadata.log.dir=/data/kafka/meta
# Metadata log segment & snapshot cadence
metadata.log.segment.bytes=1073741824
metadata.max.snapshot.interval.ms=900000

2.2 Config 2: Migration-Period Config & Health Checklist

Migrating from ZK to KRaft (3.4+ support), key configs during migration mode and migration health checklist:

# ===== ZK to KRaft: migration mode =====
# New controller node role
process.roles=controller
# Enable migration: controller dual-writes ZK and KRaft
zookeeper.metadata.migration.enable=true
# ZK connection string (still needed during migration)
zookeeper.connect=zk1:2181,zk2:2181,zk3:2181

# ===== Migration Health Checklist =====
# 1. All brokers switched to KRaft and version compliant
kafka-broker-api-versions.sh | grep KRaft
# 2. ZK no new writes: /config /brokers no longer changing
# 3. Pre-finalize full topic config audit:
kafka-configs.sh --bootstrap-server $BS \
--describe --all --topic order-event
# 4. Practiced "pull ZK": stop ZK, cluster functions normally
# 5. finalize is one-way door: confirm before executing

The checklist's five items correspond to five migration gates: all brokers switched is prerequisite for finalize (any broker still on ZK becomes orphan after finalize); ZK no new writes ensures log tail completeness; config audit prevents Incident 3's "dual-entry missed change"; pull-ZK drill is the only way to verify "truly no ZK dependency" — every hidden dependency found in drill costs 100x less than post-launch discovery.

Figure 5: Migration four phases — migration mode dual-write reversible, finalize is one-way door
Figure 5: Migration four phases — migration mode dual-write reversible, finalize is one-way door

Figure 5 · Migration four phases: migration mode dual-write reversible, finalize is one-way door

2.3 Postmortem: A Controller Failover's Chain Reaction

A real Controller failover (ZK era, scale: 60 brokers, 120k partitions):

T+0s Controller node OOM killed;

T+6s ZK ephemeral node timeout, brokers elect, broker-37 wins;

T+6s~4m50s New Controller rebuilds 120k partition metadata from ZK; broker crash detection, ISR shrink events all backlogged;

T+2m Business sees "partition leader not exists" errors — actually metadata not built, election hasn't started;

T+5m Metadata built, backlogged changes fire at once, LeaderAndIsr flood saturates brokers, recovery takes tens more seconds. Total ~5 min 40 sec.

Same scenario replayed on KRaft cluster (same scale stress test): T+2s session timeout, T+2.1s standby completes Raft election, T+2.4s replays log tail done, starts batch elections, T+4s brokers incremental sync done, client metadata refresh. Total ~4 seconds, and election is incremental broadcast not flood — the gap between two numbers is the difference between "rebuild" and "handoff". Postmortem added to SOP: ZK-era Controller OOM treated as "ten-minute event", runbook writes downgrade actions; KRaft-era treated as "second-level event", monitoring ActiveControllerCount alert downgraded from P1 to P2.

2.4 Scale Accounting: Metadata Layer Dictates Partition Ceiling

How many partitions per cluster? Bottleneck never in data layer (article 04 covered sequential write endurance), but in metadata layer. ZK-era real experience: single cluster 200k partitions requires extreme caution; Controller rebuild, full broadcast, ZK watch pressure all near limits. KRaft improves three areas: rebuild accelerated by snapshots (minutes to seconds), distribution becomes pull-based (no longer scales with broker count squared), records idempotent (retries safe). Community validated million-partition stress tests in late 3.x, but production advice remains: partitions aren't free; design by consumer group count and ordering needs; don't pre-create for "might use later" — every metadata object must be replayed on Controller failover.

2.5 Backup & Cross-DC: Metadata Layer Disaster Recovery Thinking

ZK era had awkward question: back up metadata? Theoretically partial rebuild possible, but precise recovery near impossible (which topics created later, configs changed how many rounds, no one knows). KRaft gives clean answer: @metadata log + snapshots themselves are complete metadata backup material — retain one snapshot and subsequent log segments, replay anywhere to reconstruct cluster metadata at that point. Two landing actions: metadata.log.dir on separate directory or disk (isolated from data disks, failure doesn't cascade), include snapshot files in regular backup (they're tiny, cost negligible).

Cross-DC deployment: controller quorum placement is a calculated account. 3 controllers across 3 DCs, any single DC failure leaves quorum majority alive, command system uninterrupted; cost is Raft write must cross-DC for majority ack, metadata write latency stretches from milliseconds to cross-DC RTT. Fortunately metadata writes are infrequent (only on leader election, changes), this latency cost is usually acceptable ; but broker-controller session params must widen per cross-DC RTT to avoid DC network jitter misjudged as controller crash.

Finally drills: regularly run two drills — kill active controller, pull one controller node . Former verifies failover latency within expectation (seconds), business error patterns match SOP; latter verifies quorum fault tolerance and standby health (lag zero?). During drills watch two numbers: partition recovery time P99, broker metadata incremental catch-up time. Metadata layer DR unlike data layer can't be seen directly on monitoring curves; only drills prove it.

III. Core Source Code: Four Key Locations

From 3.x core source mainline. Four pieces connect: how events enter queue, how quorum elects leader, how state replays, how brokers consume.

3.1 QuorumController: Single-Threaded Event Queue

class QuorumController implements Controller {
 // Sole event queue: all changes queue here
 private final EventQueue eventQueue;
 // Write op = first write Raft log, then mutate state machine
 CompletableFuture<Void>
 appendWriteEvent(ApiMessage record) {
 // Hand record to raftClient to append @metadata
 // After majority ack, callback applies state change
 return appendControllerRecord(record);
 }
 // Event handler serial execution: no locks, no concurrency
 // contention, Controller logic single-thread reason-able
}

Note order: log first, state machine second . This mirrors data partitions' "write log then update index" — the metadata subsystem fully replicates Kafka data path's design philosophy in its own world.

3.2 KafkaRaftClient: Quorum Internal Leader Election

// Raft layer VoteRequest handling: only vote for
// candidate with newer log (election safety)
void handleVoteRequest(VoteRequest req) {
 boolean logOk =
 req.lastLogEpoch() > myEpoch ||
 (req.lastLogEpoch() == myEpoch &&
 req.lastLogOffset() >= myEndOffset);
 if (logOk) grantVote(); else reject();
}
// Elected becomes active controller immediately,
// uncommitted log tail safely truncated to align

"Only vote for log-up-to-date candidate" is Raft's core constraint guaranteeing no committed record loss — it's the same idea as article 05's "elect leader from ISR" manifested in a different protocol: only those holding all acknowledged data are eligible to lead.

3.3 State Machine Replay: Idempotent Record Application

// Controller state = replay MetadataLog
// Each record type maps to a replay method
void replay(ApiMessageAndVersion msg) {
 if (msg is CreatePartitionsRecord) {
 // Create/expand partition: record carries full assignment info
 applyPartitionRegistration(...)
 } else if (msg is LeaderChangeRecord) {
 applyLeaderChange(...)   // election result
 } else if (msg is RemoveTopicRecord) {
 applyTopicRemoval(...)
 }
 // Replay N times same result => standby and
 // active replay same log state inevitably identical
}

3.4 BrokerMetadataListener: Incremental Consumption

// Broker side: metadata stream consumer listener
void handleMetadataDelta(MetadataDelta delta,
 MetadataImage image) {
 // IncrementalFetch pulls records after own lastOffset
 // Apply one by one to local cache
 metadataCache.update(delta, image);
 // After cache update broadcast to internal subscribers:
 // log manager / replica manager / quota manager
 publisher.publishDelta(delta);
}

Reading these four pieces together reveals KRaft invented nothing new: log, idempotency, snapshots, pull increments — all are existing Kafka data path concepts, re-enacted in the metadata layer . Understanding this, all KRaft behaviors can be deduced from articles 03/04's consumption and storage knowledge.

IV. Advanced: Seven Accounts

4.1 Election Account: Two Eras' Latency Structure

Failover latency = detection + takeover + propagation. Detection dominated by session timeout (both sides seconds, anti-jitter deliberate); takeover ZK era minutes rebuild, KRaft milliseconds replay; propagation ZK era broadcast flood, KRaft each broker self-pulls. Monitor two numbers: ActiveControllerCount (should always be 1), partition recovery time P99 after Controller switch — the latter is actual business perception.

4.2 Scale Account: Real Constraints on Partition Ceiling

Metadata layer's three amplifiers: total partitions (replay & snapshot scale), broker count (metadata distribution fan-out), change frequency (log growth rate). ZK era all three worsen with scale; KRaft leaves two-and-a-half — change frequency remains hard cost, every leader election is a log write. Capacity planning by "total partitions × change frequency" dual metric; single cluster recommended within hundreds of thousands of partitions unless you have community-level tuning capability.

4.3 Deployment Account: Combined vs Dedicated Mode

Combined mode (broker+controller same process): saves machines, suits small/medium clusters, but metadata and data traffic compete for same machine resources; dedicated mode: controllers separate 3/5 nodes, no data partitions, fault domain and load isolation, suits large clusters or frequent metadata ops. Decision rule: if Controller daily CPU exceeds 30% or metadata log growth pronounced, upgrade to dedicated mode.

4.4 Migration Account: Five Checks Before One-Way Door

Migration has four phases (Figure 5): add quorum (dual-write) → brokers switch one by one → verify → finalize. Risk concentrated around finalize: finalize is one-way door , after which any "mid-migration config tweaks" on ZK side lose sync channel. So finalize's five checks (brokers all switched, ZK no writes, config audit, pull-ZK drill, rollback drill) none skippable — especially config audit, every dual-entry manual diff is a time bomb.

4.5 Fault Domain Account: What Controller Downtime Actually Affects

During Controller unavailability (including failover window): existing partition reads/writes unaffected — data path doesn't go through Controller; affected are all "cluster-level operations": failed partitions cannot re-elect leader (leader vacancy persists until Controller returns), create/delete topic, expand/shrink partitions, config changes all halted. Thus Controller failover severity depends on how many partitions are "waiting for command" at that moment — hence metadata monitoring must be tier-1 like disk watermarks.

4.6 Monitoring Account: Five KRaft-Era Metrics

① ActiveControllerCount: each controller node reports, cluster should have exactly one 1; ② Metadata log lag: standby's replay lag behind active, sustained growth means standby unhealthy — failover degrades to rebuild; ③ Snapshot generation time & interval: bloat indicates frequent changes; ④ Broker metadata incremental latency: IncrementalFetch round-trip P99; ⑤ Controller event queue depth: backlog means single-thread strained, consider splitting batch ops or upgrading to dedicated mode.

4.7 Version Account: KRaft Timeline

2.8 introduced KRaft (early preview only); 3.3 marked production-ready (dedicated mode); 3.4 added ZK migration capability (migration mode); 3.5-3.9 continuous enhancements (dynamic quorum, migration polish); 4.0+ removes ZooKeeper , KRaft becomes sole mode. Action advice: new clusters directly on 3.8+ KRaft, stop building new ZK clusters; existing ZK clusters migrate during next major upgrade window — since upgrade needs downtime anyway, use the window to cross the one-way door.

V. High-Frequency Interview Questions (with Answers)

Q1 · What is the Controller? How is it created?

Answer: Cluster "commander", responsible for partition leader election, topic management, replica assignment, broker up/down sensing, all metadata-level ops. ZK era: broker races for /controller ephemeral node (moonlighting); KRaft era: controller quorum internal Raft election (can be separate process or co-located with broker), still exactly one active cluster-wide.

Q2 · What happens when Controller dies? Can data still be read/written?

Answer: Existing partition reads/writes completely unaffected — data path bypasses Controller. Affected are cluster-level ops: failed partition re-election pauses (leader vacancy persists), create/delete topic, expand partitions, config changes all stall until new Controller takes over. So failover window length depends on metadata rebuild speed — exactly what KRaft overhaul targets.

Q3 · What role did ZooKeeper play in old architecture?

Answer: Metadata authoritative store (broker registration, Controller election, topic/partition/config/ISR lists) + distributed coordination (ephemeral nodes, watch notifications). External CP system: Kafka relied on it for "who's alive, who does what" arbitration, but incurred dual views, slow rebuild, watch storms — three structural problems.

Q4 · Why was Controller failover slow in ZK era?

Answer: Three slow steps: detection slow (session timeout); rebuild slow — new Controller reads ZK node-by-node to rebuild in-memory view, million partitions takes minutes; propagation slow — changes via LeaderAndIsr/UpdateMetadata broadcast RPC to all brokers, large change = flood. KRaft turns latter two into "replay local log + broker self-pull", leaving only detection's inherent cost.

Q5 · Where does KRaft store metadata? What structure?

Answer: Inside controller quorum's @metadata partition — an append-only log: all metadata changes are log records, Raft-replicated, majority-committed. Controller in-memory state machine is materialized view of this log (replay rebuilds); log periodically snapshots, pre-snapshot segments cleaned. Structure isomorphic to regular data partitions, tooling shared.

Q6 · Why make metadata a log instead of a database?

Answer: Log gives four key capabilities: append-only sequential write (storage-friendly), idempotent replay (any controller restart rebuilds consistent state, standby naturally hot), snapshot truncation (replay not from zero, log bounded), incremental consumption (brokers pull increments like consuming regular topic, watch mechanism entirely retired). Database approach can't achieve "replay equals rebuild" semantics.

Q7 · How do active and standby Controllers stay consistent?

Answer: Both replay same @metadata log: active appends records and applies changes in foreground, standby continuously follows replay. Records idempotent → replay result deterministically identical. Standby slight lag normal; excessive lag must alert — otherwise failover forces it to catch up first, degrading to rebuild.

Q8 · Controller is single-threaded; won't it bottleneck?

Answer: Single-thread deliberate: state changes serialized, no locks, no concurrency bugs (ZK-era Controller's many puzzling bugs were concurrent state machine induced). Bottleneck managed by division: I/O-heavy parts (log append, Raft replication) async; event loop keeps only in-memory ops; large batch ops split into batched records. If event queue persistently backs up, scale reached dedicated mode or reduce change frequency.

Q9 · Does KRaft mode still need ZooKeeper?

Answer: No. Controller forms own quorum (controller.quorum.voters), uses Raft for election and metadata replication. 3.4+ supports ZK migration (migration mode still connects ZK during transition), 4.0+ removes ZK support entirely. Ops loses one independent component, one less fault domain, "metadata jitter amplified to cluster avalanche" accident mode eliminated.

Q10 · How to configure controller.quorum.voters? What's the semantics?

Answer: Format nodeID@host:port, comma-separated full quorum roster: e.g., [email protected]:9093,[email protected]:9093,[email protected]:9093. Each controller's node.id must appear in list. 3 or 5 nodes recommended (2N+1 tolerates N failures, Raft majority write). Note this is static list; 3.9+ supports dynamic membership change.

Q11 · How to choose combined vs dedicated mode?

Answer: Combined: broker+controller same process (process.roles=broker,controller), saves machines, suits small/medium clusters and test envs; dedicated: controllers separate 3/5 nodes, no data partitions, fault domain and load isolation, suits large clusters, frequent metadata ops (frequent elections, batch partition creates). Rule of thumb: Controller daily CPU sustained >30% → upgrade to dedicated.

Q12 · How does broker sense metadata changes in KRaft?

Answer: Broker's MetadataListener uses IncrementalFetch long-poll to pull metadata increments from active Controller (only records after own lastOffset), applies to local MetadataCache, then notifies log/replica/quota managers. Guaranteed ordering, lost data re-fetchable, uniform latency — same mechanism as consuming regular topic, replaces ZK watch.

Q13 · What does metadata snapshot do? What if no snapshots?

Answer: Snapshot is full state archive at a log offset. Role 1: restart/takeover loads snapshot then replays increments, replay time proportional to snapshot freshness; Role 2: pre-snapshot log segments cleanable, log size bounded. No snapshots → log grows unbounded, every Controller takeover becomes full replay — more metadata, slower, eventually regresses to ZK-era behavior.

Q14 · What are KRaft migration steps? Which step is irreversible?

Answer: Four phases: ① add controller quorum, enable migration mode (dual-write ZK and KRaft, ZK remains authoritative); ② brokers rolling switch to KRaft mode; ③ verify (brokers all switched, ZK no writes, config audit, pull-ZK drill); ④ finalize. finalize is one-way door: KRaft becomes sole authoritative source, no rollback after — so it must be the most deliberate step in migration plan.

Q15 · Why does migration mode easily cause "dual-config clash"?

Answer: Dual-write period has two management entrypoints: ZK-side kafka-configs and KRaft/admin API. ZK is authoritative, but editing only one side creates "single-side visible" config drift — one side appears effective, after finalize the other side's record lost, config rolls back. Countermeasure: freeze manual config changes during migration window, all changes through single entrypoint, full config diff both sides pre-finalize.

Q16 · How many nodes for controller quorum? Can it survive one down?

Answer: 3 nodes tolerate 1 failure, 5 tolerate 2 (majority alive serves). Down to no majority (e.g., 3 nodes, 2 down): active controller loses contact, cluster-level ops stall, failed partitions cannot re-elect, but existing partition reads/writes continue. So controller nodes should be cross-rack deployed, avoid same-rack failure taking quorum majority.

Q17 · After KRaft, do we still need ZK operational experience?

Answer: Most can retire: ZK session timeout tuning, watch backlog, JVM/disk tuned for ZK no longer needed. But mindset remains: session timeout anti-jitter meaning shifts to broker-controller session params; "metadata is single-point risk exposure" monitoring awareness more important — just monitoring object changes from ZK to metadata log lag and event queue.

Q18 · What does fast metadata log growth indicate? How to govern?

Answer: Indicates high cluster change frequency: frequent leader election jitter (disk slow, GC pauses causing ISR repeated shrink/expand), batch topic create/delete, partition reassignment all write logs. Governance: first dampen (article 05 disk/GC fixes), then merge batch ops (create multiple topics per request), finally tune snapshot cadence for more frequent truncation. Log growth rate itself is a leading "cluster stability" indicator.

Q19 · In KRaft mode, will creating 3,000 partitions still stall Controller?

Answer: Won't stall, but will slow — mechanism completely different: request split into batched records appended to log, async replication, Controller single-thread no longer synchronously waits each ZK write. Experience is linear slowdown not avalanche, and failures safely retryable (idempotent records). Production advice still batch-create (few hundred per batch), watch event queue depth, avoid long event-loop occupation blocking other changes.

Q20 · Reviewing a ZK cluster's KRaft migration plan, what would you check?

Answer: Follow checklist: ① version compliant (3.4+, recommend 3.7+); ② quorum deployment spec (3/5 nodes, cross-rack, dedicated metadata disk); ③ dual-write management entrypoint freeze strategy & config diff script; ④ broker rolling switch batches & rollback plan (one node per batch, confirm healthy before next); ⑤ finalize pre-five-checks executor & timing; ⑥ pull-ZK drill done? issues list; ⑦ migration monitoring switch (ActiveControllerCount, metadata log lag, event queue depth); ⑧ rollback window decision point written into change ticket — who calls stop under what conditions.

VI. One-Page Cheat Sheet

One-Sentence Architecture

Controller is cluster's sole commander (failover has standby); KRaft turns metadata into a log inside controller quorum (@metadata), state machine is just log's materialized view.

Three Mechanisms

Idempotent replay (standby hot standby & state rebuild), snapshots (replay acceleration + log bounding), IncrementalFetch (broker self-pull increments, watch retired).

Failover

Latency = session timeout (seconds, anti-jitter deliberate) + replay tail (milliseconds). ZK era was "rebuild", KRaft era is "handoff".

Migration Iron Law

Migration mode ZK is authoritative, dual-write freeze dual-entry config edits; finalize is one-way door, five pre-checks none skippable.

VII. Summary & Next Article Preview

This article compresses to one sentence: KRaft moves "command system's state" from someone else's database back into Kafka's own log. After moving back, Controller transforms from "most fragile single point" to "arbitrator ready to take over anytime" — standby hot standby, replay-is-rebuild, snapshots bounded, broker self-pull increments — four mechanisms compress failover from minutes to seconds, raise partition scale ceiling by an order of magnitude, and remove "ZK jitter amplified to cluster avalanche" from possibility list. But mind migration's one-way door: migration mode reversible, finalize then only forward — five checks aren't bureaucratic process, they're insurance paid for "irreversible". Next article returns to data path itself: Performance Tuning — assembling previous six articles' pieces (batching, compression, page cache, replication, ISR) into a whole-machine tuning methodology: how to locate bottlenecks, how to view params in layers, which params must never be touched. We'll start with a "cluster TPS won't rise but no disk bottleneck found" investigation.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

Distributed SystemsMigrationMetadataZooKeeperKafkaControllerFailoverRaftKRaft
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.