Tagged articles
Partition Assignment
12 articles · Page 1 of 1
This article analyzes Kafka's consumer group rebalance mechanisms, comparing eager and cooperative protocols, explaining why partition transfers cause consumption pauses, detailing configuration parameters like session.timeout.ms and max.poll.interval.ms, and providing troubleshooting steps for lag spikes during rolling deployments.
Configuration TuningConsumer GroupCooperative Protocol
0 likes · 22 min read

Kafka consumer issues such as message backlog, duplicate processing, and data loss often stem from consumer group rebalances triggered by changes in consumer count, partition count, subscription topics, or heartbeat and poll timeouts, and can be mitigated by tuning timeout settings, managing offset commits, and using sticky partition assignment.
Consumer GroupKafkaMessage Duplication
0 likes · 13 min read

Kafka’s internal mechanisms for leader election, partition assignment, and file storage are explained, covering how the Controller uses ZooKeeper, the ISR-based leader selection process, partition distribution strategies, segment file structures, and the evolution of offset management from Zookeeper to the __consumer_offsets topic.
KafkaLeader ElectionOffsets
0 likes · 5 min read

This article provides a comprehensive technical walkthrough of KafkaConsumer, covering Java configuration code, delivery semantics (at‑most‑once, at‑least‑once, exactly‑once), offset commit strategies, rebalance mechanisms, partition assignment algorithms, thread‑safety concerns, and internal poll implementation, followed by unrelated promotional content.
ConsumerKafkaPartition Assignment
0 likes · 16 min read

This article explains Kafka's leader election process, partition assignment strategy, distribution policies, file layout, and the evolution of consumer offset storage, providing a comprehensive overview of how Kafka ensures reliable and efficient message handling in a distributed environment.
KafkaLeader ElectionPartition Assignment
0 likes · 5 min read

This article explores strategies for batch scaling Kafka replicas with minimal data migration, presenting two design ideas, detailed calculations of broker lists, partition counts, start indexes, and replica shifts, and provides step‑by‑step algorithms and code snippets to compute optimal replica assignments for both expansion and contraction scenarios.
AlgorithmBackendKafka
0 likes · 15 min read

This article walks through the complete Kafka consumer JoinGroupRequest lifecycle, detailing how the client builds and sends the request, how the group coordinator selects the coordinator node, processes unknown and known members, elects a leader, chooses a partition assignment protocol, and transitions group states.
ConsumerGroup CoordinationJoinGroupRequest
0 likes · 26 min read

The article explains the role of Kafka's consumer group coordinator and consumer coordinator, details how group coordinators are selected, and walks through the JoinGroup, SyncGroup, LeaveGroup, and heartbeat processes, as well as partition assignment strategies and common Q&A.
Consumer GroupGroupCoordinatorJoinGroup
0 likes · 11 min read

This article explains Kafka's internal topics, preferred replicas, partition assignment processes, log directory layout, index files, offset and timestamp lookup, log retention and compaction policies, storage architecture, delayed operations, controller role, legacy consumer design flaws, rebalance workflow, and producer idempotence, providing a comprehensive overview of Kafka's backend architecture.
Consumer OffsetsIdempotenceLog Retention
0 likes · 16 min read

This article explains how Kafka determines which partition a producer sends a record to, how partition counts are configured, and how consumer groups assign partitions using the default, range, and round‑robin strategies, complemented by detailed Java code examples.
JavaKafkaPartition Assignment
0 likes · 21 min read

This article explains how Kafka consumer groups accelerate message consumption by distributing partitions across multiple consumers, details the three key characteristics of consumer groups, and provides in‑depth guidance on partition assignment strategies and offset management with practical Java code examples.
Consumer GroupKafkaPartition Assignment
0 likes · 13 min read

Kafka MirrorMaker provides near‑real‑time cross‑data‑center replication by consuming from a source cluster and producing to a target cluster, and this guide explains its core features, new vs. old consumer APIs, partition assignment strategies, performance tuning, network considerations, and practical command‑line examples.
Consumer APIKafkaMirrorMaker
0 likes · 13 min read
