How to Eliminate Duplicate and Missed Messages in Kafka Consumers

This article explains Kafka's push/pull consumption models, the impact of enable.auto.commit and auto.commit.interval.ms on offset handling, and presents practical configurations and code patterns plus MySQL and Redis based deduplication techniques to prevent both duplicate and missing message processing.

Duplicate ConsumptionIdempotencyMessage Queue

0 likes · 8 min read

How to Eliminate Duplicate and Missed Messages in Kafka Consumers

37 Interactive Technology Team

Aug 1, 2022 · Backend Development

Understanding Kafka Consumer Groups, Polling, and Offset Commit Strategies with kafka-go

The article demonstrates how kafka-go implements consumer groups by creating readers that spawn poll goroutines per partition, explains heartbeat‑driven rebalancing, and compares automatic versus manual offset‑commit strategies, highlighting their impact on throughput, reliability, and message‑processing semantics.

GoKafkaconsumer-group

0 likes · 15 min read

Understanding Kafka Consumer Groups, Polling, and Offset Commit Strategies with kafka-go

Big Data Technology & Architecture

Sep 19, 2020 · Big Data

Understanding Kafka Consumer Group Rebalance and Timeout Mechanisms

This article explains how Kafka consumer groups assign partitions, the four situations that trigger a rebalance, the impact of consumer poll timeouts, and practical ways to tune max.poll.interval.ms and max.poll.records to avoid rebalance‑related errors.

Big DataKafkaTimeout

0 likes · 12 min read

Understanding Kafka Consumer Group Rebalance and Timeout Mechanisms