Kafka Interview Guide: Core Concepts, Architecture, and Practical Tips

The article is a curated collection of Kafka interview questions and answers, intended to help candidates prepare for interviews by providing concise explanations and practical insights.

It begins with a recommendation for a beginner‑friendly Kafka book that focuses on core principles without unnecessary embellishments.

Message Queue Role and Use Cases : Improves response time by asynchronous processing, smooths traffic peaks, and reduces coupling between systems by decoupling producers and consumers.

Potential Issues of Using a Message Queue : Increases system complexity and may affect availability; consistency must be considered.

Why Choose Kafka : High throughput (millions of messages per second), distributed architecture, millisecond‑level latency, and support for both broadcast (consumer groups) and load‑balanced consumption.

Kafka Core Concepts :

Broker – a node in the Kafka cluster.

Topic – logical queue.

Partition – each topic can have multiple partitions; data and offsets are stored per partition.

Consumer Group – manages offsets per group; provides broadcast across groups and load‑balancing within a group.

Pull Consumption Model – consumers pull messages, controlling speed and batch size.

Message Storage : Messages are persisted on disk per partition in .log files, segmented for efficient access, with accompanying .index and .timeindex files for offset and timestamp lookup.

Consumer Group Rebalancing : Triggered by new consumers, consumer leave, or missed heartbeats; involves joinGroup, leader election, partition assignment, and syncGroup steps.

Guaranteeing No Message Loss :

Replication factor greater than 1 creates leader and follower replicas.

ISR (In‑Sync Replicas) list tracks replicas that are up‑to‑date.

Producer acks setting (0, 1, all) controls durability guarantees.

High Availability : Brokers register temporary nodes in ZooKeeper; the first broker becomes the controller, which monitors broker health and handles leader election for partitions.

Performance Optimizations : Zero‑copy, OS page cache, sequential disk writes, batch processing, and efficient indexing.

Key Terminology :

ISR – In‑Sync Replicas.

AR – Assigned Replicas (ISR + Out‑of‑Sync Replicas).

HW – High Watermark (minimum LEO among ISR).

LEO – Log End Offset (next write position).

LSO – Last Stable Offset (used for transactions).

LW – Low Watermark (minimum logStartOffset in AR).

Message Ordering : Order is guaranteed within a partition; to maintain order across a topic, use a single partition.

Producer Client Pipeline : Interceptor → Serializer → Partitioner → RecordAccumulator → Sender thread.

Producer Threads : Main thread builds messages; Sender thread transmits them.

Consumer Thread Safety : KafkaConsumer is not thread‑safe; use one consumer per thread or a single consumer with multiple processing threads.

Topic Management : Partition count can be increased but not decreased; choose partition count based on cluster size and required throughput.

Internal Topics : e.g., __consumer_offsets stores consumer group offsets.

Priority Replica : Preferred leader during elections.

Log Directory Structure : Each partition has a directory containing .log, .index, .timeindex, and leader‑epoch‑checkpoint files.

Offset Lookup : Use file name prefix for absolute offset, then index to find nearest entry and scan forward.

Log Retention : Configurable by time or size; supports deletion and compaction (keeping latest record per key).

Metrics to Monitor : Producer – MessagesInPerSec, BytesOutPerSec; Consumer – lag, throughput.

The article concludes with a call to like, share, and comment, emphasizing community engagement.