Tagged articles

Message Duplication

8 articles · Page 1 of 1

Jan 30, 2026 · Backend Development

Why Kafka Rebalance Causes Backlog, Duplicates, and Data Loss—and How to Fix It

Kafka consumer group rebalances can trigger message backlogs, duplicate processing, and data loss; this article explains common rebalance triggers, their impact on consumption, and practical configuration and coding strategies—such as tuning timeout parameters, using manual offset commits, and sticky partition assignment—to minimize disruptions.

Consumer GroupData lossMessage Duplication

0 likes · 12 min read

Why Kafka Rebalance Causes Backlog, Duplicates, and Data Loss—and How to Fix It

Mike Chen's Internet Architecture

Dec 27, 2025 · Backend Development

Why Kafka Messages Duplicate and How to Prevent It

The article explains the main causes of duplicate Kafka messages—including producer retries, consumer offset handling, partition leader changes, and lack of idempotence—and provides practical configuration and design solutions to achieve exactly‑once delivery.

Consumer offsetDeduplicationMessage Duplication

0 likes · 5 min read

Why Kafka Messages Duplicate and How to Prevent It

Programmer XiaoFu

Oct 16, 2025 · Operations

Stop Guessing: Kafka Message Backlog, Duplicates, and Loss Are Usually Caused by Rebalance

Kafka consumer issues such as message backlog, duplicate processing, and data loss often stem from consumer group rebalances triggered by changes in consumer count, partition count, subscription topics, or heartbeat and poll timeouts, and can be mitigated by tuning timeout settings, managing offset commits, and using sticky partition assignment.

Consumer GroupMessage DuplicationMessage Loss

0 likes · 13 min read

Stop Guessing: Kafka Message Backlog, Duplicates, and Loss Are Usually Caused by Rebalance

Programmer XiaoFu

Jul 24, 2024 · Backend Development

Root‑Cause Solutions for Kafka Message Duplicates

The article analyzes why Kafka messages are duplicated on both producer and consumer sides, explains at‑most‑once, at‑least‑once, and exactly‑once delivery semantics, and walks through three concrete remedies—idempotent producers, transactions, and consumer‑side deduplication—backed by configuration details, code samples, and practical deployment steps.

ConfigurationConsumer IdempotencyMessage Duplication

0 likes · 12 min read

Root‑Cause Solutions for Kafka Message Duplicates

Java Tech Enthusiast

Apr 30, 2024 · Backend Development

How to Prevent Message Loss and Duplication in Message Queues

This article examines why messages can be lost or duplicated in typical queue systems, explains the failure points from producer to broker to consumer, and provides practical techniques such as synchronous flushing, broker clustering, database unique constraints, and Redis deduplication to achieve reliable, idempotent processing.

Message DuplicationMessage LossMessage Queue

0 likes · 7 min read

How to Prevent Message Loss and Duplication in Message Queues

Top Architect

Mar 30, 2023 · Backend Development

Understanding Kafka Idempotent Producer and How to Prevent Message Duplicates

This article explains why message duplication occurs in Kafka, describes the three delivery semantics, and provides practical solutions—including idempotent producers, transactions, and consumer-side idempotence—along with configuration tips and code examples to achieve exactly‑once delivery.

ConfigurationIdempotenceJava

0 likes · 11 min read

Understanding Kafka Idempotent Producer and How to Prevent Message Duplicates

Top Architect

Jan 31, 2023 · Backend Development

Solving Kafka Message Duplication with Idempotent Producer, Transactions, and Consumer Idempotence

This article explains why message duplication occurs in Kafka pipelines, describes the three delivery semantics, and provides three practical solutions—Kafka idempotent producer, Kafka transactions, and consumer‑side idempotence—along with configuration tips and code examples.

Consumer IdempotenceMessage Duplicationidempotent producer

0 likes · 11 min read

Solving Kafka Message Duplication with Idempotent Producer, Transactions, and Consumer Idempotence

dbaplus Community

Mar 20, 2017 · Backend Development

How RocketMQ Tackles Message Ordering, Duplication, and Transaction Challenges

This article explains how RocketMQ, a high‑throughput distributed messaging middleware, solves the classic problems of message ordering, duplicate delivery, and transactional consistency, while detailing its load‑balancing producer, storage architecture, subscription models, and practical best‑practice recommendations.

Backend DevelopmentMessage DuplicationMessage Ordering

0 likes · 23 min read

How RocketMQ Tackles Message Ordering, Duplication, and Transaction Challenges