Key Concepts and Internal Mechanisms of Apache Kafka

What internal topics does Kafka have and what are their characteristics and purposes?

__consumer_offsets stores consumer offset information; __transaction_state stores transaction log messages.

What is a preferred replica and what special role does it play?

A preferred replica is the first replica in the AR (assigned replicas) list, ideally the partition leader. Kafka strives to evenly distribute preferred replicas across the cluster to balance leader load, known as partition balancing.

Where does Kafka perform partition allocation? Briefly describe the processes and principles.

Producer partition assignment determines the target partition for each message, which can be customized by implementing the org.apache.kafka.clients.producer.Partitioner interface.

Consumer partition assignment decides which partitions a consumer may read, configurable via the partition.assignment.strategy client property.

Replica assignment defines the placement of partition replicas across brokers when a topic is created, manually specified with the replica-assignment option of kafka-topics.sh.

Describe Kafka’s log directory structure.

Kafka stores messages per topic, each topic may have multiple partitions. Each partition corresponds to a log; logs are split into LogSegments to avoid overly large files.

Physically, a Log is a directory, each LogSegment is a data file plus two index files (and optional .txnindex files).

What index files exist in Kafka?

Each LogSegment has two index files: an offset index mapping message offsets to physical file positions, and a timestamp index mapping timestamps to offsets.

How does Kafka locate a message when an offset is specified?

Consumers call seek() after an initial poll(); the client then reads from the specified offset, handling out‑of‑range offsets according to auto.offset.reset.

How does Kafka locate a message when a timestamp is specified?

Kafka provides offsetsForTimes(), which takes a map of partitions to timestamps and returns the earliest offset whose timestamp is greater than or equal to the requested time.

Explain Kafka’s log retention mechanism.

Log retention deletes LogSegments that no longer satisfy configured policies. The broker property log.cleanup.policy defaults to “delete”.

Time‑based retention removes segments older than retentionMs (configurable via log.retention.hours/minutes/ms). Deletable segments are first removed from the in‑memory jump list, renamed with “.deleted”, and later physically deleted after file.delete.delay.ms.

Size‑based retention removes segments when the total log size exceeds retentionSize (configurable via log.retention.bytes). Segment size is limited by log.segment.bytes.

Offset‑based retention removes segments whose next segment’s base offset is less than or equal to logStartOffset.

Example: with logStartOffset = 25, segments with base offsets 0 and 11 are deletable, while the segment starting at 23 is retained.

Explain Kafka’s log compaction.

Log compaction retains only the latest record for each key. Enabling it requires setting log.cleanup.policy to “compact” and log.cleaner.enable to true.

Discuss Kafka’s underlying storage concepts.

Page Cache

The OS page cache stores disk data in memory, reducing I/O. Reads hit the cache when present; otherwise the OS loads the page. Writes modify cached pages (dirty pages) which are flushed later.

Using file‑system storage with page cache is more efficient than in‑process caches, especially for Java applications.

Zero‑Copy

Kafka uses zero‑copy (sendfile) to transfer data directly from disk to the network interface, avoiding user‑space copies and reducing context switches. In Java, FileChannel.transferTo() implements this.

Explain the principle of Kafka’s delayed operations.

Delayed operations (e.g., delayed produce, DelayedFetch, DelayedDeleteRecords) are managed by DelayedOperationPurgatory, which uses a SystemTimer based on a timing wheel to handle timeouts.

What is the role of the Kafka controller?

The controller broker manages partition and replica state, elects new leaders on failures, updates ISR changes, and handles partition reassignment when topics are expanded.

What shortcomings existed in the old Scala consumer client?

The legacy client stored consumer group metadata in ZooKeeper paths (/consumers/.../ids, /owner, /offsets). Each consumer registered watchers on these paths, leading to herd effect (massive watcher notifications) and split‑brain issues due to inconsistent state views.

What is the principle of consumer rebalancing (consumer coordinator and group coordinator)?

Rebalance is triggered by events such as new consumers joining, consumers leaving or timing out, group coordinator changes, or topic/partition count changes. The process involves four stages:

FIND_COORDINATOR – locate the GroupCoordinator broker.

JOIN_GROUP – join the group and elect a group leader.

SYNC_GROUP – leader assigns partitions according to the selected strategy and synchronizes the assignment.

HEARTBEAT – consumers send periodic heartbeats to maintain membership; missing heartbeats trigger another rebalance.

How does Kafka achieve producer idempotence?

Each producer receives a unique producer ID (PID). For each partition, the producer maintains a monotonically increasing sequence number. The broker stores the last sequence per . It accepts a message only if its sequence number equals the stored value plus one; otherwise duplicates are dropped or out‑of‑order errors are raised.

Feel free to like, bookmark, and share the article.