Comprehensive Guide to Kafka: Architecture, Performance Tuning, and Operational Practices

This document introduces Kafka as a high‑performance message queue, explaining its core benefits such as decoupling, asynchronous processing, traffic control, and zero‑copy data transfer.

Core Concepts

Kafka consists of producers, consumers, topics, partitions, and replicas. Each partition has a leader and followers, and data is stored in sequential log files (.log) with sparse indexing for fast offset lookup.

Cluster Architecture

The cluster uses a controller elected via ZooKeeper, which manages broker metadata, topic creation, and partition reassignment. Controllers and brokers communicate through ZK paths like /controller/id and /broker/ids/ .

Producer Configuration

Key producer settings include acks (0, 1, -1), batch.size , linger.ms , compression.type , and buffer.memory . Proper tuning of these parameters improves throughput while balancing latency and reliability.

props.put("acks", "-1");
props.put("batch.size", 32768);
props.put("linger.ms", 100);
props.put("compression.type", "lz4");
props.put("buffer.memory", 67108864);

Consumer Configuration

Consumers belong to a consumer group identified by group.id . Important settings include enable.auto.commit , auto.offset.reset , max.poll.records , and heartbeat parameters ( heartbeat.interval.ms , session.timeout.ms ) to ensure group coordination and rebalance handling.

props.put("group.id", "my_consumer_group");
props.put("enable.auto.commit", "true");
props.put("auto.offset.reset", "earliest");
props.put("max.poll.records", 500);

Scaling and Resource Planning

The guide outlines how to estimate required brokers, disks, memory, CPU cores, and network bandwidth for handling up to 1 billion daily requests, recommending a 5‑node physical cluster with 11 × 7 TB SAS disks, 64 GB RAM, and at least 16 CPU cores per node.

Operational Tools

Tools such as KafkaManager and Kafka‑Offset‑Monitor are introduced for managing topics, partitions, and consumer offsets. Commands for topic creation, partition reassignment, and replica factor changes are provided.

# Create a topic with 3 partitions and replication factor 2
kafka-topics.sh --create --zookeeper zk1:2181,zk2:2181,zk3:2181 \
    --replication-factor 2 --partitions 3 --topic my_topic

# Increase replication factor
kafka-reassign-partitions.sh --zookeeper zk1:2181,zk2:2181,zk3:2181 \
    --reassignment-json-file repl.json --execute

Rebalance Strategies

Kafka supports range, round‑robin, and sticky partition assignment strategies to balance load when consumers join or leave a group.

Delay Mechanism

Kafka uses a time‑wheel based delayed operation queue to handle tasks such as producer acks timeout and follower fetch retries efficiently with O(1) insertion and removal.