Master Kafka Interview Questions: Architecture, Configurations, and Best Practices

Distributed Message Middleware Overview

Kafka is a widely used distributed messaging middleware that enables asynchronous communication between services, reducing coupling and improving scalability.

Key Questions

What is a distributed message middleware?

What are its roles and use cases?

How to select a message middleware?

Kafka Basic Concepts and Architecture

Core Components

Producer : sends messages to Kafka.

Consumer : reads messages from Kafka.

Consumer Group : a set of consumers sharing the load of a topic.

Broker : Kafka server node.

Topic : logical channel for messages.

Partition : ordered log segment within a topic.

Offset : unique position of a record in a partition.

Replication : copies of partitions for high availability.

Record : the actual message (key, value, timestamp).

Topic Partition Layout

Each topic is split into multiple partitions, allowing concurrent reads and writes.

Consumer Offset Management

Consumers track their position using offsets, which can be committed automatically or manually.

Kafka Producer

The producer workflow includes configuring the client, building messages, sending them, and closing the producer.

Key configuration parameters: bootstrap.servers: broker addresses. key.serializer and value.serializer: serialization classes. acks: acknowledgment level (0, 1, -1). retries: number of send attempts. batch.num.messages: batch size for async sends. compression.type: message compression (none, gzip, snappy, lz4).

Kafka Consumer

Consumers belong to a consumer group; each partition is consumed by only one consumer within the group.

Typical consumer steps: configure client, subscribe to topics, poll messages, process, commit offsets, and close.

Important consumer settings:

bootstrap.servers

group.id

key.deserializer

and

value.deserializer

auto.offset.reset

(latest or earliest) enable.auto.commit (true/false)

max.poll.records

session.timeout.ms

Rebalance

A rebalance redistributes partitions among consumers when group membership, subscribed topics, or partition counts change. Kafka provides Range and Round‑Robin strategies, and custom assignors can be implemented.

High Availability and Performance

HA Mechanisms

Replication with In‑Sync Replicas (ISR) and Assigned Replicas (AR).

Acknowledgment configuration (acks) and retries.

Automatic leader election via ZooKeeper.

Delivery Semantics

Kafka supports at‑least‑once, at‑most‑once, and exactly‑once guarantees, controlled by producer acks and idempotent settings.

Performance Optimizations

Parallelism across partitions.

Sequential disk writes using append‑only segment files.

Page cache and OS‑level pre‑fetching.

Binary serialization and memory‑mapped files.

Lock‑free offset management and Java NIO.

Batching and compression.

Command‑Line Tools

Kafka ships with a set of scripts in the /bin directory, such as kafka-console-producer.sh, kafka-console-consumer.sh, kafka-topics.sh, kafka-consumer-groups.sh, and many others for configuration, monitoring, and administration.