Tagged articles

17 articles

Page 1 of 1

Jan 30, 2026 · Backend Development

Why Kafka Rebalance Causes Backlog, Duplicates, and Data Loss—and How to Fix It

Kafka consumer group rebalances can trigger message backlogs, duplicate processing, and data loss; this article explains common rebalance triggers, their impact on consumption, and practical configuration and coding strategies—such as tuning timeout parameters, using manual offset commits, and sticky partition assignment—to minimize disruptions.

Data lossKafkaMessage Duplication

0 likes · 12 min read

Why Kafka Rebalance Causes Backlog, Duplicates, and Data Loss—and How to Fix It

IT Services Circle

Dec 9, 2025 · Backend Development

Mastering Kafka Rebalance: Prevent Backlog, Duplicates, and Data Loss

When Kafka consumer groups rebalance, partitions are reassigned, often causing message backlog, duplicate processing, or loss; understanding the triggers, impact, and optimization techniques—like tuning timeouts, managing offset commits, and using sticky assignors—can keep your streaming pipelines reliable.

Duplicate ConsumptionKafkaOffset Management

0 likes · 13 min read

Mastering Kafka Rebalance: Prevent Backlog, Duplicates, and Data Loss

Mike Chen's Internet Architecture

Jul 9, 2025 · Backend Development

How to Prevent Duplicate Consumption in Kafka: Practical Strategies

This article explains why Kafka’s at‑least‑once delivery can cause duplicate message processing, outlines the business risks of such duplicates, and presents four practical solutions—including idempotent design, manual offset commits, exactly‑once semantics, and dead‑letter queues—to ensure reliable consumption.

Dead Letter QueueDuplicate ConsumptionExactly-Once

0 likes · 6 min read

How to Prevent Duplicate Consumption in Kafka: Practical Strategies

Top Architect

Jan 19, 2025 · Backend Development

Understanding Kafka Consumer: Offset Management, Rebalance, Partition Assignment, and Thread Safety

This article provides a comprehensive technical walkthrough of KafkaConsumer, covering Java configuration code, delivery semantics (at‑most‑once, at‑least‑once, exactly‑once), offset commit strategies, rebalance mechanisms, partition assignment algorithms, thread‑safety concerns, and internal poll implementation, followed by unrelated promotional content.

Backend DevelopmentConsumerKafka

0 likes · 16 min read

Understanding Kafka Consumer: Offset Management, Rebalance, Partition Assignment, and Thread Safety

Sanyou's Java Diary

Oct 10, 2024 · Big Data

Why Kafka Handles Millions of Messages Per Second: Inside Its High‑Performance Architecture

This article explains how Kafka achieves ultra‑high throughput and low latency despite being disk‑based, covering its Reactor I/O network model, zero‑copy techniques, partitioning strategies, segment logs with sparse indexes, sequential disk writes, page cache usage, compression, batch processing, and lock‑free offset management.

KafkaOffset ManagementReactor I/O

0 likes · 27 min read

Why Kafka Handles Millions of Messages Per Second: Inside Its High‑Performance Architecture

Architect's Guide

Apr 28, 2024 · Backend Development

Kafka Consumer Usage Example and Deep Dive into Offset Management, Rebalance, and Thread Safety

This article presents a Java Kafka consumer example, explains offset semantics (at‑most‑once, at‑least‑once, exactly‑once), details consumer rebalance mechanisms, partition assignment strategies, thread‑safety considerations, and showcases core poll, heartbeat, and auto‑commit implementations with accompanying code snippets.

ConsumerJavaKafka

0 likes · 14 min read

Kafka Consumer Usage Example and Deep Dive into Offset Management, Rebalance, and Thread Safety

JD Cloud Developers

Oct 25, 2023 · Backend Development

Master Kafka: Core Concepts, Architecture, and Practical Tips

This article explains Kafka's fundamentals, including topics, partitions, brokers, replication, producer‑consumer workflow, consumer groups, offset management, and common exception handling, while providing code examples and diagrams to help developers understand and effectively use this distributed messaging system.

Distributed SystemsJavaKafka

0 likes · 21 min read

Master Kafka: Core Concepts, Architecture, and Practical Tips

ShiZhen AI

Sep 19, 2023 · Backend Development

How RocketMQ Handles Concurrent and Ordered Message Consumption

This article explains RocketMQ's internal workflow for concurrent and ordered message consumption, covering thread‑pool configuration, pull‑message handling, request submission, execution steps, offset management, expired‑message cleanup, and answers common questions about batch retries and offset ordering.

Concurrent ConsumptionConsumeRequestMessage Queue

0 likes · 13 min read

How RocketMQ Handles Concurrent and Ordered Message Consumption

Su San Talks Tech

Mar 22, 2023 · Backend Development

Why Does RocketMQ Duplicate Message Consumption? Root Causes Explained

This article examines the various reasons why RocketMQ can deliver the same message to consumers multiple times, covering send‑retry mechanisms, consumer‑side exceptions, offset commit and persistence failures, master‑slave synchronization issues, rebalance events, and long‑processing cleanup, and offers guidance on designing idempotent solutions.

Duplicate ConsumptionMessage QueueOffset Management

0 likes · 15 min read

Why Does RocketMQ Duplicate Message Consumption? Root Causes Explained

Architecture Digest

Mar 25, 2021 · Big Data

Uber's Multi-Region Kafka Architecture and Disaster Recovery

This article explains how Uber built a multi‑region Kafka infrastructure with disaster‑recovery capabilities, detailing its replication topology, active/active and active/passive consumption modes, offset‑management service, and the challenges of ensuring reliable, low‑latency data streaming across regions.

Data StreamingKafkaOffset Management

0 likes · 9 min read

Uber's Multi-Region Kafka Architecture and Disaster Recovery

Programmer DD

Feb 20, 2021 · Big Data

How Uber Built a Multi‑Region Kafka Architecture for Disaster Recovery

Uber operates the world’s largest Kafka cluster, handling trillions of messages daily, and has engineered a multi‑region deployment with active/active and active/passive consumption modes, offset management, and uReplicator to ensure high‑availability and seamless disaster recovery across data centers.

Active-ActiveActive-PassiveKafka

0 likes · 10 min read

How Uber Built a Multi‑Region Kafka Architecture for Disaster Recovery

Big Data Technology & Architecture

Sep 18, 2020 · Big Data

Understanding Kafka Consumer Groups, Partition Assignment, and Offset Management

This article explains how Kafka consumer groups accelerate message consumption by distributing partitions across multiple consumers, details the three key characteristics of consumer groups, and provides in‑depth guidance on partition assignment strategies and offset management with practical Java code examples.

Big DataKafkaOffset Management

0 likes · 13 min read

Understanding Kafka Consumer Groups, Partition Assignment, and Offset Management

Big Data Technology & Architecture

Aug 4, 2020 · Big Data

Manual Kafka Offset Management in Spark Streaming using createDirectStream (Java & Scala)

This article explains how to use Spark Streaming's Direct Approach with Kafka, manually manage offsets, and provides complete Java and Scala implementations—including a JavaKafkaManager class, a demo application, and a Scala KafkaManager—illustrating the creation of DirectKafkaInputDStream, offset handling, and integration with Spark.

JavaKafkaOffset Management

0 likes · 14 min read

Manual Kafka Offset Management in Spark Streaming using createDirectStream (Java & Scala)

Big Data Technology & Architecture

Dec 21, 2019 · Big Data

Kafka Offset Management and Replication Mechanisms Explained

This article provides a comprehensive technical overview of Kafka's offset handling, covering the request entry point, in‑memory offset sources, offset commit and fetch implementations, file storage layout, and the leader‑follower synchronization process that ensures data replication and high‑watermark updates.

Big DataDistributed SystemsHigh Watermark

0 likes · 16 min read

Kafka Offset Management and Replication Mechanisms Explained

Architecture Digest

Aug 8, 2019 · Big Data

Kafka Practical Guide: Concepts, Architecture, Configuration, Monitoring, and Management

This article provides a comprehensive overview of Kafka, covering its basic concepts, architecture, deployment, configuration, monitoring, producer and consumer settings, offset management, high availability, replication, leader election, and practical tips for deployment, tuning, and troubleshooting in production environments.

Distributed SystemsKafkaMessage Queue

0 likes · 37 min read

Kafka Practical Guide: Concepts, Architecture, Configuration, Monitoring, and Management

MaGe Linux Operations

Dec 11, 2017 · Big Data

Master Kafka Basics: Architecture, Core Concepts, and Hands‑On Python Experiments

This article explains Kafka’s core concepts—including producers, consumers, topics, partitions, brokers, and consumer groups—describes its distributed architecture with leader‑follower replication, and provides three hands‑on kafka‑python experiments that demonstrate basic messaging, fault‑tolerant consumer groups, and offset management for reliable consumption.

Distributed StreamingKafkaOffset Management

0 likes · 9 min read

Master Kafka Basics: Architecture, Core Concepts, and Hands‑On Python Experiments

dbaplus Community

Apr 27, 2017 · Big Data

Why Kafka’s __consumer_offsets Topic Can Fill Your Disk and How to Fix It

The article explains Kafka’s default consumer offset storage mechanism, why the __consumer_offsets system topic can consume massive disk space due to frequent synchronous commits and misconfigured cleanup, and outlines practical steps to reduce offset data and enable proper log compaction.

Consumer OffsetOffset ManagementOperations

0 likes · 6 min read

Why Kafka’s __consumer_offsets Topic Can Fill Your Disk and How to Fix It