Kafka Core Concepts, Architecture, Performance Optimization, and Production Deployment Guide

Kafka provides decoupling and asynchronous processing for high‑traffic scenarios such as e‑commerce flash sales, allowing request flow to be split into risk control, inventory lock, message queue, order generation, SMS notification, and data update stages.

Core concepts include producers, consumers, topics, partitions (default one per topic, configurable), and consumer groups. A partition’s leader handles reads and writes while followers replicate data; the ISR list tracks in‑sync replicas.

Cluster architecture consists of multiple brokers, a controller node, and Zookeeper for metadata coordination. Data is stored in sequential .log files (default 1 GB per segment) and accessed via OS cache, enabling zero‑copy transfer from disk to network.

Performance techniques cover zero‑copy using Linux sendfile, sparse indexing with binary search, and tuning producer parameters such as buffer.memory , compression.type , batch.size , linger.ms , and retries to increase throughput while managing latency.

Resource evaluation for a scenario of 1 billion daily requests (≈5.5 × 10⁴ QPS peak) suggests 5 physical servers, each with 11 × 7 TB SAS disks (≈77 TB total), 64 GB RAM (128 GB optimal), and 16–32 CPU cores, plus 10 Gbps network cards.

Operational tools include KafkaManager for UI management and scripts like kafka-topics.sh --create … , kafka-reassign-partitions.sh --execute … , and JSON files for partition reassignment. Commands for increasing replication factor, moving partitions, and handling leader imbalance are also described.

Consumer‑group mechanics explain how group IDs determine partition assignment, how rebalance is coordinated via a controller, and the three rebalance strategies (range, round‑robin, sticky). Offset handling has moved from Zookeeper to the internal __consumer_offsets topic, with configurable commit policies.

Advanced topics cover custom partitioner implementation (Java code shown inside ... ), ACK settings for durability, retry handling, and the internal time‑wheel mechanism that schedules delayed operations with O(1) complexity.

Overall, the article provides a complete reference for designing, tuning, and operating a high‑availability, high‑throughput Kafka cluster in production environments.