Tagged articles
duplicate consumption
17 articles · Page 1 of 1
An inventory service deducted stock twice for the same order 34 seconds apart due to Kafka offset commit timing during rebalance; the article explains offset positions, auto vs manual commit, AckMode, max.poll.interval.ms, and idempotent consumption with database unique keys.
AckModeIdempotencyKafka
0 likes · 21 min read

The article explains why a Kafka consumer can re‑process tens of thousands of messages after a restart even though the offset was reported as successfully committed, covering auto‑commit timing, rebalance triggers, GC pauses, manual‑commit pitfalls, and practical mitigation strategies.
KafkaOFFSETRebalance
0 likes · 8 min read

The article explains why Kafka does not guarantee exactly‑once delivery and presents three practical approaches—business‑level idempotence, manual offset management, and Kafka’s transaction/EOS features—to reliably avoid duplicate message processing.
ConsumerIdempotenceKafka
0 likes · 4 min read

The article explains why RabbitMQ may deliver the same message multiple times, emphasizes that business‑level idempotency is required, and compares five concrete deduplication approaches—unique message ID with a dedup table, database unique constraints, Redis SETNX, optimistic locking, and state‑machine design—detailing their implementation, suitable scenarios, and trade‑offs.
IdempotencyOptimistic LockRabbitMQ
0 likes · 13 min read

This guide explains how to avoid duplicate message consumption in Kafka by designing unique identifiers, implementing consumer-side idempotence with deduplication tables, leveraging Kafka’s transactional features, and establishing system-level safeguards and monitoring to ensure reliable, exactly‑once processing.
IdempotenceKafkaMessage Queue
0 likes · 4 min read

To reliably handle RocketMQ's at-least-once delivery semantics, this guide explains why duplicate consumption is inevitable, outlines three defensive layers—Redis‑based idempotency, database unique constraints, and state‑machine checks—provides production‑grade Java code, and details ACK/retry strategies and monitoring practices for robust systems.
IdempotencyJavaRedis
0 likes · 9 min read

The article explains how, in a microservice architecture using RabbitMQ, you can ensure that each subscriber receives a message exactly once even when the subscriber runs multiple load‑balanced instances, by employing a fanout exchange, dedicated queues, and proper binding.
QueueRabbitMQduplicate consumption
0 likes · 6 min read

When Kafka consumer groups rebalance, partitions are reassigned, often causing message backlog, duplicate processing, or loss; understanding the triggers, impact, and optimization techniques—like tuning timeouts, managing offset commits, and using sticky assignors—can keep your streaming pipelines reliable.
Consumer GroupKafkaPerformance Tuning
0 likes · 13 min read

This article explains why Kafka’s at‑least‑once delivery can cause duplicate message processing, outlines the business risks of such duplicates, and presents four practical solutions—including idempotent design, manual offset commits, exactly‑once semantics, and dead‑letter queues—to ensure reliable consumption.
Dead Letter QueueIdempotencyKafka
0 likes · 6 min read

This article explains Kafka's push/pull consumption models, the impact of enable.auto.commit and auto.commit.interval.ms on offset handling, and presents practical configurations and code patterns plus MySQL and Redis based deduplication techniques to prevent both duplicate and missing message processing.
IdempotencyMessage Queueduplicate consumption
0 likes · 8 min read

This article explains why duplicate message consumption occurs in Kafka, outlines the underlying confirmation and consumer failure issues, and presents idempotent design strategies such as deduplication tables and code examples to ensure reliable processing.
KafkaMessage Queuebackend development
0 likes · 5 min read

This article examines the various reasons why RocketMQ can deliver the same message to consumers multiple times, covering send‑retry mechanisms, consumer‑side exceptions, offset commit and persistence failures, master‑slave synchronization issues, rebalance events, and long‑processing cleanup, and offers guidance on designing idempotent solutions.
Message QueueRebalanceRocketMQ
0 likes · 15 min read

This article explains the causes of duplicate message consumption in RabbitMQ, demonstrates how to reproduce the issue with a 10,000‑message test, and provides three practical solutions using unique IDs stored in Redis to ensure idempotent processing in Java Spring applications.
Backendduplicate consumptionmessage-queue
0 likes · 9 min read

This article explains how RocketMQ reduces message loss and duplicate consumption through synchronous sending, retry mechanisms, multi‑master brokers, synchronous disk flushing, master‑slave replication, at‑least‑once delivery, and idempotent processing techniques.
ConsumerIdempotencyMessage Reliability
0 likes · 10 min read

This article explains how message queues help decouple services, enable asynchronous processing, smooth traffic spikes, and improve system resilience, while also detailing common challenges such as reduced availability, increased complexity, duplicate consumption, data consistency, message loss, ordering, and backlog, along with practical mitigation strategies.
Data ConsistencyDecouplingMessage Queue
0 likes · 15 min read

This article analyzes the common causes of Kafka duplicate consumption, such as uncommitted offsets due to forced thread termination, auto‑commit settings, session timeouts, rebalancing, and slow processing, and provides practical solutions including disabling auto‑commit, adjusting consumer configurations, and using new consumer groups.
Consumer ConfigurationKafkaauto-commit
0 likes · 7 min read

This article reviews typical Kafka interview topics, explaining the implementation of delay queues with hierarchical time wheels, how idempotence is achieved via producer IDs, the meanings of ISR, OSR, AR, HW, LEO, LSO, strategies for guaranteeing message order, and practical solutions for handling duplicate consumption.
ISRIdempotencedelay queue
0 likes · 9 min read
