Kafka’s internal mechanisms for leader election, partition assignment, and file storage are explained, covering how the Controller uses ZooKeeper, the ISR-based leader selection process, partition distribution strategies, segment file structures, and the evolution of offset management from Zookeeper to the __consumer_offsets topic.
This article provides a comprehensive technical walkthrough of KafkaConsumer, covering Java configuration code, delivery semantics (at‑most‑once, at‑least‑once, exactly‑once), offset commit strategies, rebalance mechanisms, partition assignment algorithms, thread‑safety concerns, and internal poll implementation, followed by unrelated promotional content.
This article explains Kafka's leader election process, partition assignment strategy, distribution policies, file layout, and the evolution of consumer offset storage, providing a comprehensive overview of how Kafka ensures reliable and efficient message handling in a distributed environment.
This article explores strategies for batch scaling Kafka replicas with minimal data migration, presenting two design ideas, detailed calculations of broker lists, partition counts, start indexes, and replica shifts, and provides step‑by‑step algorithms and code snippets to compute optimal replica assignments for both expansion and contraction scenarios.
This article walks through the complete Kafka consumer JoinGroupRequest lifecycle, detailing how the client builds and sends the request, how the group coordinator selects the coordinator node, processes unknown and known members, elects a leader, chooses a partition assignment protocol, and transitions group states.
The article explains the role of Kafka's consumer group coordinator and consumer coordinator, details how group coordinators are selected, and walks through the JoinGroup, SyncGroup, LeaveGroup, and heartbeat processes, as well as partition assignment strategies and common Q&A.
This article explains Kafka's internal topics, preferred replicas, partition assignment processes, log directory layout, index files, offset and timestamp lookup, log retention and compaction policies, storage architecture, delayed operations, controller role, legacy consumer design flaws, rebalance workflow, and producer idempotence, providing a comprehensive overview of Kafka's backend architecture.
This article explains how Kafka determines which partition a producer sends a record to, how partition counts are configured, and how consumer groups assign partitions using the default, range, and round‑robin strategies, complemented by detailed Java code examples.
This article explains how Kafka consumer groups accelerate message consumption by distributing partitions across multiple consumers, details the three key characteristics of consumer groups, and provides in‑depth guidance on partition assignment strategies and offset management with practical Java code examples.
Kafka MirrorMaker provides near‑real‑time cross‑data‑center replication by consuming from a source cluster and producing to a target cluster, and this guide explains its core features, new vs. old consumer APIs, partition assignment strategies, performance tuning, network considerations, and practical command‑line examples.
