BestHub
Sign in
Tagged articles
12 articles
Page 1 of 1
Architect Chen
Architect Chen
Jan 2, 2026 · Backend Development

Preventing Duplicate Consumption in Kafka: Design, Idempotence, and Configuration Strategies

This guide explains how to avoid duplicate message consumption in Kafka by designing unique identifiers, implementing consumer-side idempotence with deduplication tables, leveraging Kafka’s transactional features, and establishing system-level safeguards and monitoring to ensure reliable, exactly‑once processing.

Backend DevelopmentDuplicate ConsumptionExactly-Once
0 likes · 4 min read
Preventing Duplicate Consumption in Kafka: Design, Idempotence, and Configuration Strategies
Ray's Galactic Tech
Ray's Galactic Tech
Dec 20, 2025 · Backend Development

Production-Ready Idempotency for RocketMQ Duplicate Consumption (Full Code)

To reliably handle RocketMQ's at-least-once delivery semantics, this guide explains why duplicate consumption is inevitable, outlines three defensive layers—Redis‑based idempotency, database unique constraints, and state‑machine checks—provides production‑grade Java code, and details ACK/retry strategies and monitoring practices for robust systems.

Duplicate ConsumptionIdempotencyJava
0 likes · 9 min read
Production-Ready Idempotency for RocketMQ Duplicate Consumption (Full Code)
IT Services Circle
IT Services Circle
Dec 9, 2025 · Backend Development

Mastering Kafka Rebalance: Prevent Backlog, Duplicates, and Data Loss

When Kafka consumer groups rebalance, partitions are reassigned, often causing message backlog, duplicate processing, or loss; understanding the triggers, impact, and optimization techniques—like tuning timeouts, managing offset commits, and using sticky assignors—can keep your streaming pipelines reliable.

Duplicate ConsumptionKafkaOffset Management
0 likes · 13 min read
Mastering Kafka Rebalance: Prevent Backlog, Duplicates, and Data Loss
Mike Chen's Internet Architecture
Mike Chen's Internet Architecture
Jul 9, 2025 · Backend Development

How to Prevent Duplicate Consumption in Kafka: Practical Strategies

This article explains why Kafka’s at‑least‑once delivery can cause duplicate message processing, outlines the business risks of such duplicates, and presents four practical solutions—including idempotent design, manual offset commits, exactly‑once semantics, and dead‑letter queues—to ensure reliable consumption.

Dead Letter QueueDuplicate ConsumptionExactly-Once
0 likes · 6 min read
How to Prevent Duplicate Consumption in Kafka: Practical Strategies
Senior Tony
Senior Tony
Feb 1, 2025 · Backend Development

How to Eliminate Duplicate and Missed Messages in Kafka Consumers

This article explains Kafka's push/pull consumption models, the impact of enable.auto.commit and auto.commit.interval.ms on offset handling, and presents practical configurations and code patterns plus MySQL and Redis based deduplication techniques to prevent both duplicate and missing message processing.

Duplicate ConsumptionIdempotencyMessage Queue
0 likes · 8 min read
How to Eliminate Duplicate and Missed Messages in Kafka Consumers
Mike Chen's Internet Architecture
Mike Chen's Internet Architecture
Jul 3, 2024 · Backend Development

Preventing Duplicate Message Consumption in Kafka

This article explains why duplicate message consumption occurs in Kafka, outlines the underlying confirmation and consumer failure issues, and presents idempotent design strategies such as deduplication tables and code examples to ensure reliable processing.

Backend DevelopmentDuplicate ConsumptionKafka
0 likes · 5 min read
Preventing Duplicate Message Consumption in Kafka
Su San Talks Tech
Su San Talks Tech
Mar 22, 2023 · Backend Development

Why Does RocketMQ Duplicate Message Consumption? Root Causes Explained

This article examines the various reasons why RocketMQ can deliver the same message to consumers multiple times, covering send‑retry mechanisms, consumer‑side exceptions, offset commit and persistence failures, master‑slave synchronization issues, rebalance events, and long‑processing cleanup, and offers guidance on designing idempotent solutions.

Duplicate ConsumptionMessage QueueOffset Management
0 likes · 15 min read
Why Does RocketMQ Duplicate Message Consumption? Root Causes Explained
Top Architect
Top Architect
Sep 29, 2022 · Backend Development

Handling Duplicate Consumption in RabbitMQ: Scenarios, Tests, and Solutions

This article explains the causes of duplicate message consumption in RabbitMQ, demonstrates how to reproduce the issue with a 10,000‑message test, and provides three practical solutions using unique IDs stored in Redis to ensure idempotent processing in Java Spring applications.

BackendDuplicate Consumptionmessage-queue
0 likes · 9 min read
Handling Duplicate Consumption in RabbitMQ: Scenarios, Tests, and Solutions
IT Architects Alliance
IT Architects Alliance
Feb 27, 2022 · Backend Development

Why Message Queues Matter: Decoupling, Asynchrony, and Real‑World Pitfalls

This article explains how message queues help decouple services, enable asynchronous processing, smooth traffic spikes, and improve system resilience, while also detailing common challenges such as reduced availability, increased complexity, duplicate consumption, data consistency, message loss, ordering, and backlog, along with practical mitigation strategies.

Backend ArchitectureData ConsistencyDecoupling
0 likes · 15 min read
Why Message Queues Matter: Decoupling, Asynchrony, and Real‑World Pitfalls
Big Data Technology & Architecture
Big Data Technology & Architecture
Jul 15, 2020 · Big Data

Root Causes and Solutions for Kafka Duplicate Consumption

This article analyzes the common causes of Kafka duplicate consumption, such as uncommitted offsets due to forced thread termination, auto‑commit settings, session timeouts, rebalancing, and slow processing, and provides practical solutions including disabling auto‑commit, adjusting consumer configurations, and using new consumer groups.

Consumer ConfigurationDuplicate ConsumptionKafka
0 likes · 7 min read
Root Causes and Solutions for Kafka Duplicate Consumption
Full-Stack Internet Architecture
Full-Stack Internet Architecture
Jul 4, 2020 · Big Data

Common Kafka Interview Questions: Delay Queues, Idempotence, ISR/AR, HW/LEO/LSO, Message Ordering, and Duplicate Consumption

This article reviews typical Kafka interview topics, explaining the implementation of delay queues with hierarchical time wheels, how idempotence is achieved via producer IDs, the meanings of ISR, OSR, AR, HW, LEO, LSO, strategies for guaranteeing message order, and practical solutions for handling duplicate consumption.

Duplicate ConsumptionISRIdempotence
0 likes · 9 min read
Common Kafka Interview Questions: Delay Queues, Idempotence, ISR/AR, HW/LEO/LSO, Message Ordering, and Duplicate Consumption