BestHub
Sign in
Tagged articles
34 articles
Page 1 of 1
Su San Talks Tech
Su San Talks Tech
Jan 30, 2026 · Backend Development

Why Kafka Rebalance Causes Backlog, Duplicates, and Data Loss—and How to Fix It

Kafka consumer group rebalances can trigger message backlogs, duplicate processing, and data loss; this article explains common rebalance triggers, their impact on consumption, and practical configuration and coding strategies—such as tuning timeout parameters, using manual offset commits, and sticky partition assignment—to minimize disruptions.

Data lossKafkaMessage Duplication
0 likes · 12 min read
Why Kafka Rebalance Causes Backlog, Duplicates, and Data Loss—and How to Fix It
Tech Freedom Circle
Tech Freedom Circle
Jan 15, 2026 · Backend Development

Kafka Rebalance Storm Crushed 120k QPS in JD Interview – How to Understand and Fix

In a JD senior Java architect interview, a Kafka consumer‑group rebalance storm caused QPS to drop from 120k to zero, triggering massive message loss and latency spikes, and the article walks through the rebalance fundamentals, failure causes, impact analysis, cooperative sticky assignor migration, and comprehensive monitoring and mitigation strategies.

Distributed SystemsKafkaconsumer-group
0 likes · 28 min read
Kafka Rebalance Storm Crushed 120k QPS in JD Interview – How to Understand and Fix
IT Services Circle
IT Services Circle
Dec 9, 2025 · Backend Development

Mastering Kafka Rebalance: Prevent Backlog, Duplicates, and Data Loss

When Kafka consumer groups rebalance, partitions are reassigned, often causing message backlog, duplicate processing, or loss; understanding the triggers, impact, and optimization techniques—like tuning timeouts, managing offset commits, and using sticky assignors—can keep your streaming pipelines reliable.

Duplicate ConsumptionKafkaOffset Management
0 likes · 13 min read
Mastering Kafka Rebalance: Prevent Backlog, Duplicates, and Data Loss
mikechen
mikechen
Sep 11, 2025 · Backend Development

How Does Kafka Guarantee Message Order? Key Mechanisms Explained

This article explains why message ordering is critical in scenarios such as financial transactions, e‑commerce order updates, audit logs, and IoT devices, then details Kafka’s core components—topics, partitions, offsets, producers, brokers, and consumer groups—and describes how single‑partition consumption and consistent key hashing ensure ordered processing while balancing throughput.

BackendKafkaMessage Ordering
0 likes · 5 min read
How Does Kafka Guarantee Message Order? Key Mechanisms Explained
Instant Consumer Technology Team
Instant Consumer Technology Team
Jun 5, 2025 · Big Data

Mastering Kafka in Production: Boost Throughput, Ensure Reliability, and Avoid Data Loss

This article shares practical Kafka production insights, covering architecture overview, producer throughput tuning, message loss prevention, broker and consumer configurations, duplicate consumption avoidance, backlog mitigation, ordering guarantees, and the mechanics of consumer group rebalancing, helping engineers build stable, high‑performance streaming pipelines.

Big DataKafkaMessage Queue
0 likes · 15 min read
Mastering Kafka in Production: Boost Throughput, Ensure Reliability, and Avoid Data Loss
Sohu Tech Products
Sohu Tech Products
Mar 12, 2025 · Databases

Understanding Redis Streams: Core Commands and SpringBoot Integration

The article introduces Redis Streams as a Kafka‑like messaging structure, explains its fundamental concepts and seven core commands (XADD, XRANGE, XREAD, XGROUP CREATE, XREADGROUP, XACK, XTRIM), demonstrates integration with Spring Boot, and evaluates its fit for lightweight, low‑backlog queue scenarios.

Redis CommandsSpringBootStream
0 likes · 14 min read
Understanding Redis Streams: Core Commands and SpringBoot Integration
MaGe Linux Operations
MaGe Linux Operations
Oct 8, 2023 · Big Data

Understanding Kafka: Core Concepts, Architecture, and Performance Secrets

This article explains Kafka’s fundamental role as a message system, detailing topics, partitions, producers, consumers, replica management, consumer groups, the controller, Zookeeper coordination, and performance optimizations such as sequential writes, zero‑copy, log segmentation, and network design, providing a comprehensive overview for big‑data practitioners.

Big DataDistributed SystemsKafka
0 likes · 11 min read
Understanding Kafka: Core Concepts, Architecture, and Performance Secrets
Efficient Ops
Efficient Ops
Oct 7, 2023 · Big Data

Master Kafka Basics: Topics, Partitions, Producers, and Cluster Architecture

This article explains Kafka's role as a messaging system, covering core concepts such as topics, partitions, producers, consumers, messages, cluster architecture, replicas, consumer groups, controller coordination with Zookeeper, and performance optimizations like sequential writes and zero‑copy networking.

Big DataDistributed SystemsKafka
0 likes · 11 min read
Master Kafka Basics: Topics, Partitions, Producers, and Cluster Architecture
Sanyou's Java Diary
Sanyou's Java Diary
Jun 26, 2023 · Big Data

Master Kafka Interview Questions: Architecture, Partitioning, and Reliability Explained

This article provides a comprehensive overview of Kafka, covering its core architecture, message queue models, communication process, partition selection, consumer groups, rebalancing strategies, partition assignment algorithms, reliability guarantees, replica synchronization, and reasons for removing Zookeeper in newer versions.

KafkaPartitioningReliability
0 likes · 20 min read
Master Kafka Interview Questions: Architecture, Partitioning, and Reliability Explained
Architect's Guide
Architect's Guide
Jun 6, 2023 · Backend Development

Kafka Core Concepts, Architecture, Performance Tuning, and Cluster Capacity Planning

This article provides a comprehensive overview of Kafka, covering its core value for decoupling and asynchronous processing, fundamental concepts such as producers, consumers, topics, partitions and replication, high‑performance mechanisms like zero‑copy and OS cache, detailed resource evaluation for CPU, memory, disk and network, operational tools, consumer‑group rebalance strategies, LEO/HW offsets, controller management, and delayed‑task scheduling.

BackendCluster PlanningKafka
0 likes · 29 min read
Kafka Core Concepts, Architecture, Performance Tuning, and Cluster Capacity Planning
ShiZhen AI
ShiZhen AI
Oct 25, 2022 · Operations

How to Diagnose Unexpected Errors When Adding a New Kafka Consumer Group

When starting a new Kafka consumer group, an unexpected SyncGroup error occurs due to a RecordTooLargeException, and the article walks through log inspection, identifies the oversized __consumer_offsets record, and resolves the issue by increasing the message.max.bytes configuration.

KafkaRecordTooLargeExceptionSyncGroup
0 likes · 5 min read
How to Diagnose Unexpected Errors When Adding a New Kafka Consumer Group
ShiZhen AI
ShiZhen AI
Sep 27, 2022 · Big Data

What Is a Kafka Consumer Group Coordinator?

The article explains the role of Kafka's consumer group coordinator and consumer coordinator, details how group coordinators are selected, and walks through the JoinGroup, SyncGroup, LeaveGroup, and heartbeat processes, as well as partition assignment strategies and common Q&A.

GroupCoordinatorJoinGroupKafka
0 likes · 11 min read
What Is a Kafka Consumer Group Coordinator?
37 Interactive Technology Team
37 Interactive Technology Team
Aug 1, 2022 · Backend Development

Understanding Kafka Consumer Groups, Polling, and Offset Commit Strategies with kafka-go

The article demonstrates how kafka-go implements consumer groups by creating readers that spawn poll goroutines per partition, explains heartbeat‑driven rebalancing, and compares automatic versus manual offset‑commit strategies, highlighting their impact on throughput, reliability, and message‑processing semantics.

GoKafkaconsumer-group
0 likes · 15 min read
Understanding Kafka Consumer Groups, Polling, and Offset Commit Strategies with kafka-go
MaGe Linux Operations
MaGe Linux Operations
Jun 19, 2022 · Big Data

Visualizing Kafka: Core Concepts Explained with Diagrams

This article provides a diagram‑driven walkthrough of Kafka’s fundamental concepts—including topics, partitions, producers, consumers, consumer groups, and cluster architecture—explaining how messages flow, are stored, and achieve reliability and ordering within a distributed streaming system.

Cluster ArchitectureKafkaPartitions
0 likes · 6 min read
Visualizing Kafka: Core Concepts Explained with Diagrams
Java Interview Crash Guide
Java Interview Crash Guide
Feb 10, 2022 · Backend Development

Why Kafka Consumer Rebalance Stops Your Apps and How to Fix It

This article explains Kafka consumer group rebalance, covering core concepts, trigger conditions, detailed protocol steps, common pitfalls like long pause times, and modern improvements such as static membership and incremental cooperative rebalance, plus practical configuration tips to minimize disruptions.

BackendIncremental RebalanceStatic Membership
0 likes · 13 min read
Why Kafka Consumer Rebalance Stops Your Apps and How to Fix It
IT Architects Alliance
IT Architects Alliance
Nov 15, 2021 · Industry Insights

Why Kafka’s Topic‑Partition Design Powers Scalable Messaging

This article explains Kafka’s core architecture—including topics, partitions, replication, consumer groups, controller coordination with Zookeeper, and performance tricks like sequential writes and zero‑copy—to show how it achieves high‑throughput, fault‑tolerant messaging for large‑scale systems.

Distributed SystemsKafkaMessage Queue
0 likes · 11 min read
Why Kafka’s Topic‑Partition Design Powers Scalable Messaging
ByteDance Dali Intelligent Technology Team
ByteDance Dali Intelligent Technology Team
Aug 26, 2021 · Backend Development

Design and Implementation of a Distributed KV‑Based Message Queue

This article explains the core concepts and detailed design of a custom message queue built on a distributed key‑value store, covering terminology, architecture, broker metadata, topic metadata, message format, sending, storage, retrieval, delay handling, retry mechanisms, dead‑letter queues, and TTL policies.

Delay MessageDistributed KVMessage Queue
0 likes · 13 min read
Design and Implementation of a Distributed KV‑Based Message Queue
IT Architects Alliance
IT Architects Alliance
May 11, 2021 · Big Data

Demystifying Kafka: Core Concepts of Topics, Partitions, and Architecture

This article provides a clear, visual walkthrough of Kafka’s fundamental architecture, explaining how producers and consumers interact, the role of topics and partitions, consumer groups, and ZooKeeper’s coordination, helping readers grasp message flow, storage, ordering, and fault‑tolerance in a distributed streaming system.

KafkaMessage QueuePartition
0 likes · 6 min read
Demystifying Kafka: Core Concepts of Topics, Partitions, and Architecture
IT Architects Alliance
IT Architects Alliance
Apr 24, 2021 · Big Data

Kafka Basics and Cluster Architecture Overview

This article explains Kafka's role as a decoupling message buffer, describes topics, partitions, replication, consumer groups, controller coordination with Zookeeper, and performance optimizations such as sequential writes, zero‑copy, log segmentation, and its reactor‑style network design.

MessagingPartitionReplication
0 likes · 10 min read
Kafka Basics and Cluster Architecture Overview
JavaEdge
JavaEdge
Dec 3, 2020 · Backend Development

Kafka Partition vs Consumer Group: Which Consumption Mode to Use?

This article explains how Kafka clients can request data via RPC, compares the partition consumption mode with the consumer‑group mode, outlines their protocols, server‑side processing, advantages, drawbacks, and provides guidance on when to choose each approach.

backend-developmentconsumer-groupconsumption-mode
0 likes · 4 min read
Kafka Partition vs Consumer Group: Which Consumption Mode to Use?
Full-Stack Internet Architecture
Full-Stack Internet Architecture
Oct 22, 2020 · Backend Development

Core Concepts and Architecture of RocketMQ

This article introduces RocketMQ’s core concepts, including its deployment architecture, naming server, broker and client roles, subscription model, consumption modes, queue allocation algorithms, rebalancing, offset storage, transaction and delayed messages, as well as filtering mechanisms, providing a solid foundation for further practice.

BackendDistributed SystemsMessage Queue
0 likes · 13 min read
Core Concepts and Architecture of RocketMQ
Big Data Technology & Architecture
Big Data Technology & Architecture
Sep 18, 2020 · Big Data

Understanding Kafka Consumer Groups, Partition Assignment, and Offset Management

This article explains how Kafka consumer groups accelerate message consumption by distributing partitions across multiple consumers, details the three key characteristics of consumer groups, and provides in‑depth guidance on partition assignment strategies and offset management with practical Java code examples.

Big DataKafkaOffset Management
0 likes · 13 min read
Understanding Kafka Consumer Groups, Partition Assignment, and Offset Management
Programmer DD
Programmer DD
Nov 27, 2018 · Backend Development

How to Prevent Duplicate Message Consumption with Spring Cloud Stream Consumer Groups

This article explains why duplicate message consumption occurs when using Spring Cloud Stream with RabbitMQ or Kafka, introduces the concept of consumer groups, and provides a step‑by‑step Java example showing how to configure and use consumer groups to ensure each message is processed by only one instance.

Kafkaconsumer-groupspring-boot
0 likes · 6 min read
How to Prevent Duplicate Message Consumption with Spring Cloud Stream Consumer Groups
Qunar Tech Salon
Qunar Tech Salon
Feb 27, 2018 · Big Data

Root Cause Analysis of Kafka Consumer Group Coordinator Failure and __consumer_offsets Compaction Issues

The article investigates a Kafka cluster outage where several brokers became unavailable and consumers could not join groups, explains the role of __consumer_offsets, analyzes the coordinator selection logic, identifies a stuck loadGroupsForPartition operation and compact thread failure, and documents the recovery steps taken.

Log CompactionOFFSETconsumer-group
0 likes · 11 min read
Root Cause Analysis of Kafka Consumer Group Coordinator Failure and __consumer_offsets Compaction Issues
Programmer DD
Programmer DD
Feb 4, 2018 · Backend Development

How to Enable Message Partitioning in Spring Cloud Stream for Consistent Consumer Routing

This guide explains how to configure message partitioning in Spring Cloud Stream by adjusting consumer and producer properties—such as group, destination, partitioned flag, instanceCount, instanceIndex, partitionKeyExpression, and partitionCount—to ensure that messages with the same characteristics are consistently processed by the same instance across multiple service instances.

ConfigurationMessage PartitioningMicroservices
0 likes · 4 min read
How to Enable Message Partitioning in Spring Cloud Stream for Consistent Consumer Routing
Programmer DD
Programmer DD
Jan 22, 2018 · Cloud Native

How Spring Cloud Stream Abstracts Messaging Middleware with Binders and Channels

This article explains how Spring Cloud Stream uses binders to isolate applications from messaging middleware, defines channels for input and output, demonstrates publish‑subscribe with RabbitMQ, shows how consumer groups prevent duplicate processing, and introduces message partitioning for ordered handling.

BinderChannelMessage Partitioning
0 likes · 11 min read
How Spring Cloud Stream Abstracts Messaging Middleware with Binders and Channels
MaGe Linux Operations
MaGe Linux Operations
Dec 11, 2017 · Big Data

Master Kafka Basics: Architecture, Core Concepts, and Hands‑On Python Experiments

This article explains Kafka’s core concepts—including producers, consumers, topics, partitions, brokers, and consumer groups—describes its distributed architecture with leader‑follower replication, and provides three hands‑on kafka‑python experiments that demonstrate basic messaging, fault‑tolerant consumer groups, and offset management for reliable consumption.

Distributed StreamingKafkaOffset Management
0 likes · 9 min read
Master Kafka Basics: Architecture, Core Concepts, and Hands‑On Python Experiments