Tagged articles
offset management
21 articles · Page 1 of 1
The article examines a real payment loss incident caused by Kafka's default automatic offset commit, explains why automatic commits become a hidden trap at high traffic, and provides a step‑by‑step guide to switching to manual commits with async‑first, sync‑fallback, idempotency, dead‑letter handling, batch strategies, and concurrency controls for reliable ten‑million QPS consumption.
IdempotencyKafkaMessage Queue
0 likes · 15 min read

The article analyzes why single‑threaded message consumption hits processing, I/O/CPU mismatch, and fault‑tolerance limits, then walks through a step‑by‑step evolution—pull/worker split, key‑based routing, sliding‑window offset commits, backpressure, and multi‑layer parallelism—to achieve stable ten‑million‑QPS throughput.
Message Queuebackpressureconsumer parallelism
0 likes · 22 min read

The article explains why Kafka does not guarantee exactly‑once delivery and presents three practical approaches—business‑level idempotence, manual offset management, and Kafka’s transaction/EOS features—to reliably avoid duplicate message processing.
ConsumerIdempotenceKafka
0 likes · 4 min read

Kafka consumer group rebalances can trigger message backlogs, duplicate processing, and data loss; this article explains common rebalance triggers, their impact on consumption, and practical configuration and coding strategies—such as tuning timeout parameters, using manual offset commits, and sticky partition assignment—to minimize disruptions.
Consumer GroupKafkaMessage Duplication
0 likes · 12 min read

When Kafka consumer groups rebalance, partitions are reassigned, often causing message backlog, duplicate processing, or loss; understanding the triggers, impact, and optimization techniques—like tuning timeouts, managing offset commits, and using sticky assignors—can keep your streaming pipelines reliable.
Consumer GroupKafkaPerformance Tuning
0 likes · 13 min read

Kafka consumer issues such as message backlog, duplicate processing, and data loss often stem from consumer group rebalances triggered by changes in consumer count, partition count, subscription topics, or heartbeat and poll timeouts, and can be mitigated by tuning timeout settings, managing offset commits, and using sticky partition assignment.
Consumer GroupKafkaMessage Duplication
0 likes · 13 min read

This article explains why Kafka’s at‑least‑once delivery can cause duplicate message processing, outlines the business risks of such duplicates, and presents four practical solutions—including idempotent design, manual offset commits, exactly‑once semantics, and dead‑letter queues—to ensure reliable consumption.
Dead Letter QueueIdempotencyKafka
0 likes · 6 min read

This article provides a comprehensive technical walkthrough of KafkaConsumer, covering Java configuration code, delivery semantics (at‑most‑once, at‑least‑once, exactly‑once), offset commit strategies, rebalance mechanisms, partition assignment algorithms, thread‑safety concerns, and internal poll implementation, followed by unrelated promotional content.
ConsumerKafkaPartition Assignment
0 likes · 16 min read

This article explains how Kafka achieves ultra‑high throughput and low latency despite being disk‑based, covering its Reactor I/O network model, zero‑copy techniques, partitioning strategies, segment logs with sparse indexes, sequential disk writes, page cache usage, compression, batch processing, and lock‑free offset management.
KafkaReactor I/Ooffset management
0 likes · 27 min read

This article presents a Java Kafka consumer example, explains offset semantics (at‑most‑once, at‑least‑once, exactly‑once), details consumer rebalance mechanisms, partition assignment strategies, thread‑safety considerations, and showcases core poll, heartbeat, and auto‑commit implementations with accompanying code snippets.
ConsumerJavaKafka
0 likes · 14 min read

This article explains Kafka's fundamentals, including topics, partitions, brokers, replication, producer‑consumer workflow, consumer groups, offset management, and common exception handling, while providing code examples and diagrams to help developers understand and effectively use this distributed messaging system.
JavaKafkaMessage Queue
0 likes · 21 min read

This article explains RocketMQ's internal workflow for concurrent and ordered message consumption, covering thread‑pool configuration, pull‑message handling, request submission, execution steps, offset management, expired‑message cleanup, and answers common questions about batch retries and offset ordering.
Concurrent ConsumptionConsumeRequestMessage Queue
0 likes · 13 min read

This article examines the various reasons why RocketMQ can deliver the same message to consumers multiple times, covering send‑retry mechanisms, consumer‑side exceptions, offset commit and persistence failures, master‑slave synchronization issues, rebalance events, and long‑processing cleanup, and offers guidance on designing idempotent solutions.
Message QueueRebalanceRocketMQ
0 likes · 15 min read

This article explains how Uber built a multi‑region Kafka infrastructure with disaster‑recovery capabilities, detailing its replication topology, active/active and active/passive consumption modes, offset‑management service, and the challenges of ensuring reliable, low‑latency data streaming across regions.
Data StreamingKafkadisaster recovery
0 likes · 9 min read

Uber operates the world’s largest Kafka cluster, handling trillions of messages daily, and has engineered a multi‑region deployment with active/active and active/passive consumption modes, offset management, and uReplicator to ensure high‑availability and seamless disaster recovery across data centers.
Active-ActiveActive-PassiveKafka
0 likes · 10 min read

This article explains how Kafka consumer groups accelerate message consumption by distributing partitions across multiple consumers, details the three key characteristics of consumer groups, and provides in‑depth guidance on partition assignment strategies and offset management with practical Java code examples.
Consumer GroupKafkaPartition Assignment
0 likes · 13 min read

This article explains how to use Spark Streaming's Direct Approach with Kafka, manually manage offsets, and provides complete Java and Scala implementations—including a JavaKafkaManager class, a demo application, and a Scala KafkaManager—illustrating the creation of DirectKafkaInputDStream, offset handling, and integration with Spark.
JavaKafkaScala
0 likes · 14 min read

This article provides a comprehensive technical overview of Kafka's offset handling, covering the request entry point, in‑memory offset sources, offset commit and fetch implementations, file storage layout, and the leader‑follower synchronization process that ensures data replication and high‑watermark updates.
High WatermarkKafkaReplication
0 likes · 16 min read

This article provides a comprehensive overview of Kafka, covering its basic concepts, architecture, deployment, configuration, monitoring, producer and consumer settings, offset management, high availability, replication, leader election, and practical tips for deployment, tuning, and troubleshooting in production environments.
KafkaMessage Queuedistributed systems
0 likes · 37 min read

This article explains Kafka’s core concepts—including producers, consumers, topics, partitions, brokers, and consumer groups—describes its distributed architecture with leader‑follower replication, and provides three hands‑on kafka‑python experiments that demonstrate basic messaging, fault‑tolerant consumer groups, and offset management for reliable consumption.
Consumer GroupKafkaProducer-Consumer
0 likes · 9 min read

The article explains Kafka’s default consumer offset storage mechanism, why the __consumer_offsets system topic can consume massive disk space due to frequent synchronous commits and misconfigured cleanup, and outlines practical steps to reduce offset data and enable proper log compaction.
Operationsbig-dataconsumer offset
0 likes · 6 min read
