Skip to content
Kafka Interview Deep Dive: 8 Core Questions on ISR, Rebalance, Zero-Copy & Message Guarantees

Kafka Interview Deep Dive: 8 Core Questions on ISR, Rebalance, Zero-Copy & Message Guarantees

This comprehensive guide covers Kafka architecture, ISR mechanism, message loss prevention, duplicate consumption solutions, Rebalance protocols, zero-copy/Page Cache performance internals, and a decision tree for choosing between Kafka, RocketMQ, and RabbitMQ.

Rain's Path to Java Mastery
Rain's Path to Java Mastery
Rain's Path to Java Mastery

Kafka Architecture: Producer, Broker, Consumer, ZooKeeper/KRaft

Producer: How to Send Messages Reliably

1. Partition Strategy

Default sticky partitioner sends same batch to one partition to reduce network overhead. With a key, uses hash(key) % partition. Partition determines ordering : only messages within the same partition are ordered.

2. acks Mechanism

acks=0

: Fire-and-forget, highest throughput, highest data loss risk. acks=1: Leader writes successfully then returns; sufficient for most scenarios. acks=all/-1: Leader + all ISR replicas must acknowledge; most reliable but higher latency .

3. Idempotence & Transactions

Enable enable.idempotence=true; Producer adds Producer ID and sequence numbers, Broker deduplicates for exactly-once semantics. Combined with transactions, achieves atomic cross-partition writes.

Broker: Storage & Replication Core

Log Storage

Each partition is a segmented log file on disk, append-only, old segments roll by time or size. Kafka's high throughput stems from sequential I/O and zero-copy.

Replica Mechanism

Leader handles all reads/writes.

Followers asynchronously pull from Leader; those keeping up stay in ISR.

If Leader fails, Controller elects new Leader from ISR.

Controller

Special Broker managing partition/replica state and Leader election, elected via ZooKeeper or KRaft.

Consumer: Consumer Groups & Offset Management

Consumers work as consumer groups . Core rules: one partition consumed by only one consumer in group (preserves partition order); if consumers > partitions, some idle; membership changes trigger Rebalance , pausing consumption — avoid frequent rebalances.

Offset Management

Old versions stored offsets in ZooKeeper; new versions use internal topic __consumer_offsets. Consumers can commit manually or automatically. Trade-off: process-then-commit risks duplicates; commit-then-process risks loss. Interview favorite.

Coordination Layer: ZooKeeper vs KRaft

Legacy: ZooKeeper Mode

Manages cluster metadata (live brokers, topic configs).

Elects Controller.

Stored consumer offsets (deprecated since 0.9+).

Drawbacks: extra cluster ops cost, Zab protocol becomes bottleneck at scale.

Modern: KRaft Mode (Kafka 2.8+, production-ready 3.x)

Embeds metadata management in Brokers using Raft protocol to elect Quorum Controller.

Metadata log separate from data log; Controller maintains its own metadata partition.

Benefits: simplified deployment, faster recovery (seconds), supports million partitions, vastly more topics per cluster.

Topic, Partition, Replica Mechanics

Topic = logical category; Partition = physical shard (parallelism engine); Replica = backup for HA.

Partition: Parallelism Engine

Strict ordering within a partition; no ordering across partitions.

Consumer count ≤ partition count for parallelism.

Key hashing routes same key to same partition.

Partition count determines max parallelism — core reason for Kafka's high throughput.

Replica: Backup Vault

Leader serves all read/write; Followers only sync.

ISR (In-Sync Replicas) : replicas caught up with Leader; only ISR members eligible for Leader election.

ACK mechanism: producer chooses acks=0/1/-1 to decide how many replicas must acknowledge.

Example: Partition 0 with 3 replicas across 3 brokers. Broker1=Leader, Broker2/3=Followers. If Broker1 dies, Broker2 promoted, service recovers in seconds.

Interview Hot Questions

Why no read/write separation?

Followers only sync, don't serve reads — ensures consistency, avoids stale reads, simplifies design, better performance.

More partitions = better?

No. Excess partitions increase metadata overhead, client connections, recovery time. Recommend ≤1000 partitions per broker.

ISR vs AR difference?

AR (Assigned Replicas) = all assigned replicas; ISR = subset of AR that are caught up with Leader. Lagging followers kicked out after replica.lag.time.max.ms (default 30s), rejoin when caught up.

Four Pillars of High Throughput

1. Sequential Disk Write

Traditional DBs use random I/O (seek ~10ms). Kafka partitions are append-only logs — sequential writes. Mechanical disk sequential write ~600MB/s, near memory speed.

Random Write: ≈100KB/s–few MB/s

Sequential Write: ≥600MB/s

2. Zero-Copy

Traditional send: 4 copies + 2 context switches. Kafka uses FileChannel.transferTo() → sendfile syscall: data moves directly from Page Cache to NIC, bypassing user space. Cuts copies by 50%, eliminates context switches, doubles throughput.

fileChannel.transferTo(position, count, socketChannel)

3. Page Cache

Kafka avoids JVM heap cache; relies on OS Page Cache. All reads/writes hit Page Cache first; async flush to disk. Consumers read hot data from Page Cache — disk becomes backup. Avoids GC overhead, survives process restarts, OS manages hot/cold automatically.

GC Overhead: JVM Heap Cache = High (Full GC risk); OS Page Cache = None

Process Restart: JVM Heap Cache = Cache lost; OS Page Cache = Cache retained

Hot/Cold Swap: JVM Heap Cache = Manual; OS Page Cache = OS auto

Memory Efficiency: JVM Heap Cache = Low (object headers); OS Page Cache = High

4. Batch Compression

Producer batches messages ( linger.ms wait, batch.size limit), compresses whole batch (Snappy/LZ4/ZSTD/Gzip). Broker stores compressed batch without decompression; consumer decompresses on fetch. Reduces network packets, disk usage, end-to-end compression.

ISR (In-Sync Replicas) Mechanism

ISR = Leader + Followers caught up within replica.lag.time.max.ms (default 30s). Since 0.9, only time-based lag check (removed message-count lag) to avoid flapping.

LEO & HW

LEO (Log End Offset): Last message offset + 1 on each replica; maintained by each replica.

HW (High Watermark): Min LEO across all ISR replicas; consumers can read up to HW; Leader computes & syncs to Followers.

Process: Followers fetch → update LEO → Leader computes HW as min(ISR LEOs) → if Follower lags >30s, removed from ISR; rejoins when caught up.

ISR & Producer ACK Binding

props.put("acks", "all");
props.put("min.insync.replicas", 2);
acks=0

: Fire-and-forget → 🟥 High loss risk acks=1: Leader ack only → 🟨 Medium (Leader crash loses data) acks=all/-1: All ISR replicas ack → 🟩 Very low (with min.insync.replicas)

With acks=all, message must be written by all current ISR replicas. If ISR shrinks to only Leader but min.insync.replicas=2, producer gets exception — refuses write to guarantee durability . CAP trade-off: prefer unavailability over data loss.

Failover: Why Elect from ISR?

ISR replicas have identical (or near-identical) data → no loss on failover. If all ISR down, unclean.leader.election.enable decides: true = allow OSR (out-of-sync) Leader → data loss but service restored; false (default) = partition unavailable until ISR recovers. New Leader truncates logs beyond its HW for consistency.

Production Recommendations

acks=all

+ min.insync.replicas≥2 (e.g., Spring Boot: spring.kafka.producer.acks=-1, min.insync.replicas=2).

Monitor ISR shrinkage alerts — indicates broker/network/disk issues.

Tune replica.lag.time.max.ms for traffic spikes (30s usually fine).

Never enable unclean.leader.election unless data loss acceptable.

Message Loss Scenarios & Fixes

Producer Side

Async send without waiting for ack loses data on network/broker failure. Even get() with acks=0/1 loses data if Leader crashes before replication.

Fix: Max ack + Retry + Idempotence

props.put("acks", "all");
props.put("retries", Integer.MAX_VALUE);
props.put("max.in.flight.requests.per.connection", 5);
props.put("enable.idempotence", true);
producer.send(record, (metadata, exception) -> {
    if (exception != null) { /* log/alert */ }
});

Broker Side

Loss when: replication.factor=1; min.insync.replicas=1 with acks=all (Leader alone counts as ISR); unclean.leader.election.enable=true.

Fix: Cluster Insurance

replication.factor=3
min.insync.replicas=2
unclean.leader.election.enable=false

Math: with N replicas, M min-insync, tolerate N-M broker failures without loss. N=3, M=2 is sweet spot.

Consumer Side

Auto-commit loses messages: consumer polls batch, processes halfway, crashes before next auto-commit interval → offset already committed → unprocessed messages skipped forever.

Fix: Manual Offset Commit

props.put("enable.auto.commit", "false");
while (true) {
    ConsumerRecords<String,String> records = consumer.poll(Duration.ofMillis(100));
    for (ConsumerRecord<String,String> record : records) {
        process(record);
    }
    consumer.commitSync(); // block until success
}

Best practice: sync+async combo — async during normal loop, sync in finally block on shutdown/exception. Core principle: process first, then commit; consumer logic must be idempotent.

Duplicate Consumption & Idempotence

Kafka guarantees At Least Once — no loss, but duplicates possible.

Three Root Causes

Producer retry: retries>0 without idempotence; broker stored but ACK lost → resend → duplicate.

Auto-commit offset: processed but not yet committed → crash → re-consume on restart.

Rebalance: partition reassignment before offset commit → new consumer re-processes.

Idempotence Solutions (Priority Order)

DB Unique Key : Message ID + business key as PK. Simple, strong consistency. DB dependency, overhead. Most business scenarios.

Redis Distributed Lock : SETNX with message ID key. Fast, distributed. Lock expiry, weak consistency. High concurrency, relaxed consistency.

Kafka Idempotent Producer : enable.idempotence=true. Auto dedup at broker. Single partition/session only. Fixes producer-side duplicates.

Kafka Transactions : transactional.id config. Cross-partition atomicity. Slow, complex. Cross-system strong consistency.

Business State Machine : State transitions guard reprocessing. Natural, no extra cost. Requires stateful design. Clear state flows.

Consumer-Side Pattern: Dedup Table + Manual Commit

-- Dedup table
CREATE TABLE event_dedupe (
    event_id VARCHAR(64) PRIMARY KEY,
    create_time TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

// Consumer logic
public void process(ConsumerRecord<String,String> record) {
    String eventId = extractEventId(record.value());
    int rows = jdbcTemplate.update(
        "INSERT IGNORE INTO event_dedupe (event_id) VALUES (?)", eventId);
    if (rows > 0) {
        doBusinessLogic(record.value());
    } else {
        log.warn("Duplicate intercepted: {}", eventId);
    }
}

Redis: SET key value NX EX <ttl>. Must use manual commit ( enable.auto.commit=false), process and DB insert in same transaction, key = message ID + business ID. Clean dedup table via daily TTL job (retention > max message replay window). If no business ID, use topic-partition-offset but prefer upstream global ID (snowflake, traceId).

Rebalance Mechanism & Consumer Group Coordination

What Triggers Rebalance?

Group membership change (join/leave/crash).

Subscribed topics change.

Partition count change for subscribed topics.

During Rebalance, entire group pauses consumption — major performance pain point.

Group Coordinator

Broker-side component per group. First consumer finds it via FindCoordinator. Coordinator manages members, drives Rebalance, assigns partitions, stores offsets in __consumer_offsets.

Incremental Rebalance Protocol (Kafka 0.11+)

JoinGroup : All consumers report to Coordinator; one elected Group Leader (not Controller).

SyncGroup : Coordinator sends member list to Leader; Leader computes assignment via PartitionAssignor; Coordinator distributes assignments.

Assignment Strategies

RangeAssignor (default): Per-topic partition ranges. Best for single topic.

RoundRobinAssignor: Round-robin across all partitions. Best for multiple topics, balanced load.

StickyAssignor: Preserve existing assignment. Reduces partition movement.

CooperativeStickyAssignor: Incremental cooperative rebalance. Recommended for production .

Optimizations

1. Frequent Rebalance Fixes

Increase session.timeout.ms (default 10s) and max.poll.interval.ms (default 5min).

Decrease max.poll.records (default 500).

Optimize processing logic to avoid long blocks.

2. Static Membership

Set unique group.instance.id per consumer → becomes static member. On restart, Coordinator waits session.timeout.ms before Rebalance; if rejoins within window, no Rebalance triggered.

3. Cooperative (Incremental) Rebalance

Classic Eager protocol: all consumers stop, full reassignment. CooperativeStickyAssignor: only affected consumers revoke changed partitions; others continue. Two-phase: first revoke, then assign — most partitions uninterrupted.

Behavior: Eager = All stop, full reshuffle 🛑; Cooperative = Only changed partitions move ✅

Partition Pause: Eager = All partitions briefly unavailable; Cooperative = Most partitions unaffected

Complexity: Eager = Low; Cooperative = Higher, multi-step

Best For: Eager = Stable small groups; Cooperative = Large-scale, high-stability

Kafka vs RocketMQ vs RabbitMQ Selection

One-Line Positioning

RabbitMQ: Lightweight, high-reliability traditional broker; complex routing.

RocketMQ: Alibaba financial-grade; balances performance & reliability; feature-rich for Chinese internet.

Kafka: De facto standard for big data ecosystem; extreme throughput; log collection, stream processing.

Core Comparison

Language: RabbitMQ=Erlang, RocketMQ=Java, Kafka=Scala/Java

Positioning: RabbitMQ=Enterprise message broker, RocketMQ=Financial-grade distributed MQ, Kafka=Distributed streaming platform

Throughput: RabbitMQ=10K/sec, RocketMQ=100K/sec, Kafka=1M+/sec

Latency: RabbitMQ=Microsecond, RocketMQ=Millisecond, Kafka=Millisecond

Reliability: RabbitMQ=Very high (transactions), RocketMQ=Very high (financial transactions), Kafka=High (configurable)

Message Model: RabbitMQ=Exchange+Queue (flexible routing), RocketMQ=Topic+Queue (broadcast/cluster), Kafka=Topic+Partition (sharding)

Retry: RabbitMQ=Dead-letter queue, RocketMQ=Multi-level retry + DLQ, Kafka=DIY

Delayed Msg: RabbitMQ=Plugin needed, RocketMQ=Native arbitrary precision, Kafka=Not supported

Transaction Msg: RabbitMQ=Weak support, RocketMQ=Native strong support, Kafka=No (external coordination)

Ops Complexity: RabbitMQ=Low, RocketMQ=Medium, Kafka=High

Ecosystem: RabbitMQ=Mature (multi-language), RocketMQ=Strong in China, Kafka=Huge (big data)

Architecture Differences

RabbitMQ: Exchange is core — direct/topic/fanout/headers routing.

RocketMQ: Stateless NameServer registry; Broker master-slave HA.

Kafka: Partition is core — multiple partitions per topic enable parallelism.

Decision Tree

Volume > 100M/day → Kafka (sequential I/O, zero-copy built for massive data).

Medium volume but need: delayed messages, distributed transactions, strict ordering → RocketMQ (native features like half-message transaction, arbitrary delay).

Lower volume, complex routing, multi-language, low ops overhead → RabbitMQ (flexible exchange, great admin UI).

Interview Follow-Ups

Why Kafka throughput so high?

Sequential disk write (avoids random I/O).

Zero-copy via sendfile (kernel-to-NIC).

Partition parallelism (multiple consumers per topic).

When NOT to use Kafka?

Strict ordering with high concurrency (only per-partition ordering).

Transactional messaging (Kafka transactions limited vs RocketMQ).

Delayed/scheduled messages (no native support).

Why RocketMQ over RabbitMQ in China?

Java stack — easy to debug/extend.

Proven at Alibaba Double-11 (trillion messages).

10x throughput of RabbitMQ.

Active Chinese community, rich docs.

RocketMQ Transaction Message Strong Consistency?

Half-message mechanism: send invisible half-message → execute local transaction → commit/rollback → if no response, broker polls producer for status → guarantees eventual consistency.

RabbitMQ Backpressure Pain?

Queue indexes in memory; massive pile-up to disk causes index bloat → memory/CPU thrashing → throughput collapses. Not a "reservoir"; use as agile "pipe" with flow control.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

KafkaRabbitMQRocketMQZero-CopyInterview PreparationPartitioningKRaftISRRebalanceMessage Delivery Guarantees
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.