Tagged articles
Rebalance
34 articles · Page 1 of 1
This article analyzes Kafka's consumer group rebalance mechanisms, comparing eager and cooperative protocols, explaining why partition transfers cause consumption pauses, detailing configuration parameters like session.timeout.ms and max.poll.interval.ms, and providing troubleshooting steps for lag spikes during rolling deployments.
Configuration TuningConsumer GroupCooperative Protocol
0 likes · 22 min read

An inventory service deducted stock twice for the same order 34 seconds apart due to Kafka offset commit timing during rebalance; the article explains offset positions, auto vs manual commit, AckMode, max.poll.interval.ms, and idempotent consumption with database unique keys.
AckModeIdempotencyKafka
0 likes · 21 min read

The article explains why Kafka consumers can reprocess tens of thousands of messages after a restart despite successful offset commits, detailing the roles of automatic commit timing, rebalance triggers, GC pauses, and the limitations of manual commits, and offers practical configuration and idempotent processing solutions.
ConsumerJavaKafka
0 likes · 8 min read

The article explains why a Kafka consumer can re‑process tens of thousands of messages after a restart even though the offset was reported as successfully committed, covering auto‑commit timing, rebalance triggers, GC pauses, manual‑commit pitfalls, and practical mitigation strategies.
KafkaOFFSETRebalance
0 likes · 8 min read

At the 10‑million‑QPS scale, expanding a message‑queue cluster no longer hinges on simply adding brokers; it requires coordinated online upgrades of metadata, data migration with dynamic throttling, cooperative consumer rebalance, shadow‑traffic warm‑up, and rollback snapshots, making the act of adding machines the hardest part.
10M QPSMessage QueueRebalance
0 likes · 27 min read

At a 5,000‑instance, 20,000‑partition Kafka consumer cluster, the default eager Rebalance caused a 90‑second stop‑the‑world pause, illustrating how a single GC‑induced heartbeat miss can flood brokers with tens of millions of messages; the article dissects eager vs cooperative protocols, built‑in assignor limits, and four custom assignor patterns to scale Rebalance from 100 K to tens of millions of QPS.
AssignorCooperative ProtocolCustom Scheduler
0 likes · 28 min read

Even when a consumer process runs and logs normally, Kafka may deem it dead and trigger a rebalance because the poll interval exceeds max.poll.interval.ms, a situation known as ‘false dead’; this article explains the root cause and practical ways to prevent it.
ConsumerKafkaMultithreading
0 likes · 8 min read

Even when a consumer thread continues processing and logging normally, Kafka may still consider it dead and trigger a rebalance because the poll interval exceeds the configured timeout, a situation known as “fake dead”.
ConsumerKafkaRebalance
0 likes · 7 min read

The paper introduces ReBalance, a training‑free two‑stage inference control framework that uses model confidence signals to dynamically balance reasoning depth, achieving up to a 10‑point accuracy gain and a 35.4% reduction in token length across multiple LLM sizes and benchmarks.
Balanced ThinkingConfidence SteeringICLR 2026
0 likes · 9 min read

The article explains why merely “knowing how to use” Kafka is insufficient, detailing how offset commits, consumer acknowledgments, producer acks, and rebalance behavior affect reliability, and provides concrete code examples, risk scenarios, and configuration recommendations to prevent message loss and duplicate processing in production systems.
IdempotenceKafkaMessage Reliability
0 likes · 7 min read

Kafka consumer group rebalances can trigger message backlogs, duplicate processing, and data loss; this article explains common rebalance triggers, their impact on consumption, and practical configuration and coding strategies—such as tuning timeout parameters, using manual offset commits, and sticky partition assignment—to minimize disruptions.
Consumer GroupKafkaMessage Duplication
0 likes · 12 min read

In a JD senior Java architect interview, a Kafka consumer‑group rebalance storm caused QPS to drop from 120k to zero, triggering massive message loss and latency spikes, and the article walks through the rebalance fundamentals, failure causes, impact analysis, cooperative sticky assignor migration, and comprehensive monitoring and mitigation strategies.
Consumer GroupKafkaRebalance
0 likes · 28 min read

When Kafka consumer groups rebalance, partitions are reassigned, often causing message backlog, duplicate processing, or loss; understanding the triggers, impact, and optimization techniques—like tuning timeouts, managing offset commits, and using sticky assignors—can keep your streaming pipelines reliable.
Consumer GroupKafkaPerformance Tuning
0 likes · 13 min read

Kafka consumer issues such as message backlog, duplicate processing, and data loss often stem from consumer group rebalances triggered by changes in consumer count, partition count, subscription topics, or heartbeat and poll timeouts, and can be mitigated by tuning timeout settings, managing offset commits, and using sticky partition assignment.
Consumer GroupKafkaMessage Duplication
0 likes · 13 min read

This article deeply explores Kafka consumer group rebalance mechanisms, identifies performance pitfalls of frequent rebalances, and provides a comprehensive set of configuration tweaks, assignment strategies, batch processing techniques, and monitoring practices to achieve a more stable and high‑throughput Kafka consumer system.
BackendConsumerKafka
0 likes · 16 min read

This article shares hands‑on production experience with Kafka, covering how to tune producer throughput, prevent message loss, control duplicate consumption, handle backlog, maintain ordering, understand rebalance, and explains why Kafka delivers such high performance.
ConfigurationConsumerKafka
0 likes · 15 min read

This article provides a comprehensive technical walkthrough of KafkaConsumer, covering Java configuration code, delivery semantics (at‑most‑once, at‑least‑once, exactly‑once), offset commit strategies, rebalance mechanisms, partition assignment algorithms, thread‑safety concerns, and internal poll implementation, followed by unrelated promotional content.
ConsumerKafkaPartition Assignment
0 likes · 16 min read

This article explains Kafka consumer semantics such as at‑most‑once, at‑least‑once and exactly‑once delivery, shows how to configure and commit offsets, describes the rebalance process and partition‑assignment strategies, and discusses thread‑safety and task scheduling with illustrative Java code examples.
ConsumerKafkaOFFSET
0 likes · 17 min read

This article presents a Java Kafka consumer example, explains offset semantics (at‑most‑once, at‑least‑once, exactly‑once), details consumer rebalance mechanisms, partition assignment strategies, thread‑safety considerations, and showcases core poll, heartbeat, and auto‑commit implementations with accompanying code snippets.
ConsumerJavaKafka
0 likes · 14 min read

This article provides a comprehensive guide to KafkaConsumer, covering message delivery semantics (at‑most‑once, at‑least‑once, exactly‑once), practical exactly‑once implementations, consumer rebalance triggers and strategies, partition assignment algorithms, thread‑safety considerations, and detailed Java code examples of the consumer workflow.
ConsumerJavaKafka
0 likes · 14 min read

This article examines the various reasons why RocketMQ can deliver the same message to consumers multiple times, covering send‑retry mechanisms, consumer‑side exceptions, offset commit and persistence failures, master‑slave synchronization issues, rebalance events, and long‑processing cleanup, and offers guidance on designing idempotent solutions.
Message QueueRebalanceRocketMQ
0 likes · 15 min read

This article provides a comprehensive overview of Kafka consumer group rebalance, covering version compatibility, rebalance triggers, assignment strategies, generation handling, protocol details, the full rebalance workflow, listener usage, and complete Java code examples for offset management with database integration.
ConsumerGroupJavaKafka
0 likes · 19 min read

This article walks through the complete Kafka consumer JoinGroupRequest lifecycle, detailing how the client builds and sends the request, how the group coordinator selects the coordinator node, processes unknown and known members, elects a leader, chooses a partition assignment protocol, and transitions group states.
ConsumerGroup CoordinationJoinGroupRequest
0 likes · 26 min read

The article walks through the Kafka consumer heartbeat lifecycle, detailing how the HeartbeatThread is started, paused, and used to send heartbeat and LeaveGroup requests, how the GroupCoordinator validates and processes these requests, the client’s response handling, and the resulting state transitions illustrated with diagrams.
ConsumerGroupCoordinatorKafka
0 likes · 14 min read

This article presents a detailed overview of Apache Spark's Adaptive Query Execution evolution, its optimization techniques, and performance gains, followed by an in‑depth discussion of Apache Kyuubi's architecture, security integrations, cloud‑native capabilities, and practical Rebalance + Z‑Order strategies that enhance data‑warehouse task efficiency and query performance.
Apache SparkBig Data OptimizationKyuubi
0 likes · 19 min read

This article explains Kafka consumer group rebalance, covering core concepts, trigger conditions, detailed protocol steps, common pitfalls like long pause times, and modern improvements such as static membership and incremental cooperative rebalance, plus practical configuration tips to minimize disruptions.
BackendConsumer GroupIncremental Rebalance
0 likes · 13 min read

This article provides a comprehensive technical overview of Apache Kafka, covering its distributed messaging architecture, key features such as high‑throughput read/write, replication, partitioning, consumer group mechanics, offset management, rebalance processes, and practical code examples for synchronous and asynchronous offset commits.
Consumer OffsetsDistributed MessagingKafka
0 likes · 22 min read

This article explains how Kafka consumer groups assign partitions, the four situations that trigger a rebalance, the impact of consumer poll timeouts, and practical ways to tune max.poll.interval.ms and max.poll.records to avoid rebalance‑related errors.
Consumer GroupKafkaRebalance
0 likes · 12 min read

This article explains how Kafka consumer parameters like session.timeout.ms, heartbeat.interval.ms, and max.poll.interval.ms affect group rebalance, consumer liveness, and real‑time alert processing in high‑availability microservice architectures.
ConsumerKafkaRebalance
0 likes · 15 min read

This article explains the fundamentals of Apache Kafka's consumer group rebalance protocol, covering partitioning, group coordination, JoinGroup and SyncGroup requests, heartbeat handling, common pitfalls, and best‑practice warnings for reliable streaming data processing.
Consumer GroupRebalancedistributed-systems
0 likes · 13 min read

The article examines a Kafka consumer rebalance error caused by exceeding max.poll.interval.ms, explains the underlying mechanics of poll intervals, offset handling, and provides practical solutions such as adjusting max.poll.interval.ms, limiting poll records, and committing offsets per message to prevent frequent rebalances.
JavaKafkaRebalance
0 likes · 9 min read

This article dives deep into Kafka’s internal mechanics, explaining how brokers replicate data, how the controller coordinates the cluster via ZooKeeper, the roles of leader and follower replicas, ISR management, request handling, fail‑over strategies, and consumer group rebalancing, all illustrated with diagrams.
BackendBroker ControllerISR
0 likes · 36 min read

The article explains Kafka's consumer‑group rebalance and Pulsar's unified queue/stream subscription models, compares their partition assignment strategies, and demonstrates both with Docker‑based Pulsar setups, Java consumer code, and practical failover and exclusive scenarios.
Consumer GroupFailoverKafka
0 likes · 6 min read

This article explains Kafka’s High‑Level and Low‑Level consumer models, the semantics of Consumer Groups, the rebalance algorithm and its drawbacks, and outlines the planned redesign in Kafka 0.9.x that introduces a central Coordinator to solve herd and split‑brain issues.
ConsumerHigh-Level ConsumerKafka
0 likes · 20 min read
