Tagged articles
exactly-once
77 articles · Page 1 of 1
This article explains exactly-once processing semantics in data engineering, contrasting it with at-most-once and at-least-once, detailing why retries cause duplicates, the roles of checkpoints, idempotency, and transactions, why technical exactly-once doesn't guarantee business correctness, and the four critical failure tests needed before production deployment.
Data ConsistencyIdempotencycheckpoint
0 likes · 16 min read


java1234Oct 26, 2026 · Backend Development
Kafka 4.2 introduces Share Groups (KIP-932) allowing multiple consumers to process records from the same partition concurrently using fetch locks and explicit acknowledgments (ACCEPT, RELEASE, REJECT, RENEW), while Kafka Streams gains server-side rebalancing (KIP-1071) and dead-letter queues (KIP-1034); version 4.2.2 adds 33 fixes including exactly-once checkpoint cleanup, deadlock resolution, and latency metric corrections.
Dead Letter QueueKIP-1034KIP-1071
0 likes · 17 min read


Niu LiuSep 25, 2026 · Backend Development
This article explains that Exactly Once semantics in message queues like Kafka only guarantee exactly-once processing within the messaging system's transaction boundary, not for external side effects like payment APIs, and provides practical patterns using database unique constraints, idempotency keys, and timeout handling to achieve true end-to-end exactly-once behavior.
IdempotencyKafkaPayment Processing
0 likes · 9 min read


ITPUBSep 14, 2026 · Big Data
This article compares Flink CDC with traditional Canal-based architectures, explains Flink CDC's core principles including lock-free snapshots and exactly-once semantics, provides DataStream and SQL code examples for MySQL integration, and covers five high-frequency interview questions on DDL handling, DELETE capture, and large-table optimization.
DebeziumFlink CDCFlink SQL
0 likes · 12 min read

This article details how to build a production-ready real-time data warehouse using Flink, covering CDC ingestion, event-time processing, dimension joins, upsert aggregation, state TTL, checkpoint tuning, exactly-once verification, and a pre-launch checklist for reliability.
CDCDimension JoinEvent Time
0 likes · 22 min read

This article provides a comprehensive guide to integrating Apache Kafka Streams with Spring Boot for production-grade stream processing, covering core concepts, state storage with RocksDB, window aggregation strategies, exactly-once semantics, interactive queries, multi-tenancy isolation, and operational best practices.
Interactive QueriesKafka StreamsMulti-tenancy
0 likes · 16 min read

This article details migrating from a legacy Canal+Kafka+Flink+Hive stack to a modern Flink CDC + Iceberg real-time data lake, covering Spring Boot orchestration, chunk-based full/incremental sync, Interval Join for wide tables, automated schema evolution, hidden partitioning, Exactly-Once checkpoint tuning, and Kubernetes deployment practices.
Apache IcebergFlink CDCKubernetes
0 likes · 17 min read

This guide details how to achieve end-to-end exactly-once semantics with Apache Flink and Apache Paimon, covering consistency models, three-layer coordination mechanisms, production-grade configurations, concurrency rules, verification tests, failure injection scenarios, performance trade-offs, and operational SOPs for lakehouse streaming workloads.
Apache FlinkApache PaimonProduction Operations
0 likes · 26 min read

Alibaba Cloud Message Queue for Kafka introduces a native message‑to‑lake capability that integrates Apache Iceberg with OSS Table Bucket, eliminating Spark/Flink/Kafka Connect, providing exactly‑once semantics, automatic schema management, dual write modes, smart partitioning, and up to ten‑fold performance gains across diverse real‑time analytics scenarios.
Apache IcebergCloud NativeKafka
0 likes · 12 min read

This article explains how Kafka maintains order within partitions using keys and offsets, why ordering across partitions can break, and how to preserve strict sequencing through producer configuration, idempotent producers, and exactly‑once transactional processing.
KafkaTransactionsdata ordering
0 likes · 9 min read

The article explains why Kafka does not guarantee exactly‑once delivery and presents three practical approaches—business‑level idempotence, manual offset management, and Kafka’s transaction/EOS features—to reliably avoid duplicate message processing.
ConsumerIdempotenceKafka
0 likes · 4 min read

This article demystifies Kafka's ordering guarantees, explains the difference between partition‑level, key‑level, and global ordering, and provides a complete, production‑grade guide—including architecture, configuration, code samples, and best‑practice patterns—to achieve reliable ordered processing while maintaining high throughput and scalability.
CDCConsumerIdempotence
0 likes · 35 min read

This article explains practical strategies to avoid duplicate message consumption in Kafka, covering business idempotency with unique IDs, database or Redis deduplication tables, enabling producer idempotence, consumer-side checks, and Kafka's transaction-based exactly‑once semantics, along with their trade‑offs and suitable scenarios.
BackendIdempotencyKafka
0 likes · 4 min read

This article presents a comprehensive, end‑to‑end solution for preventing duplicate consumption in RabbitMQ by combining architecture design, routing strategies, idempotent controls, state machines, and ACK mechanisms, and details six concrete implementation options with code examples.
IdempotencyJavaRabbitMQ
0 likes · 9 min read

This guide explains how to avoid duplicate message consumption in Kafka by designing unique identifiers, implementing consumer-side idempotence with deduplication tables, leveraging Kafka’s transactional features, and establishing system-level safeguards and monitoring to ensure reliable, exactly‑once processing.
Backend DevelopmentIdempotenceKafka
0 likes · 4 min read

This guide explains how to enable idempotence in Kafka producers, configure essential settings, understand its guarantees and limitations, and when to adopt transactions or external deduplication for cross‑partition and exactly‑once semantics.
External DeduplicationJavaKafka
0 likes · 4 min read

This guide explains how to configure Spring Boot and Kafka to achieve Exactly-Once semantics, covering core concepts, Maven dependencies, YAML settings, sample code for producers and consumers, execution flow, and advanced tips for reliable transactional messaging.
Backendexactly-oncespring-boot
0 likes · 8 min read

This article explains Kafka's idempotent producer feature, describing how enabling idempotence ensures that duplicate messages caused by retries or network issues are deduplicated so that each message is persisted only once in a partition, and outlines the underlying PID and sequence number mechanism.
Idempotenceexactly-oncemessage-queue
0 likes · 4 min read

This article explains why Kafka’s at‑least‑once delivery can cause duplicate message processing, outlines the business risks of such duplicates, and presents four practical solutions—including idempotent design, manual offset commits, exactly‑once semantics, and dead‑letter queues—to ensure reliable consumption.
Dead Letter QueueIdempotencyKafka
0 likes · 6 min read

This article explains common causes of duplicate message consumption in high‑traffic systems and presents a three‑layer defense—producer idempotence, broker de‑duplication, and consumer idempotent design—plus monitoring and reconciliation strategies to achieve reliable exactly‑once processing.
IdempotenceKafkaRocketMQ
0 likes · 7 min read

This article explains what Flink’s Exactly‑Once semantics actually guarantee, why it does not mean each event is processed only once, how checkpointing and two‑phase commit sinks enable end‑to‑end exactly‑once, and the three safeguards needed for true exactly‑once computation.
Flinkbig datacheckpointing
0 likes · 5 min read

This article explains how Apache InLong provides automatic, secure, high‑performance real‑time data transfer to StarRocks, detailing the transactional Stream Load API, the two‑phase commit process, Flink‑based ingestion architecture, exactly‑once guarantees, and performance test results across different parallelism levels.
InLongStarRocksbig data
0 likes · 11 min read

This article breaks down Kafka’s message delivery guarantees—at‑most‑once, at‑least‑once, and exactly‑once—explaining the underlying producer, broker, and consumer requirements, configuration settings, and practical code snippets to achieve reliable delivery.
At-Least-OnceBackend DevelopmentKafka
0 likes · 5 min read

This article provides a detailed overview of Apache Kafka, covering its distributed streaming architecture, storage mechanisms, replication, consumer groups, compression techniques, exactly‑once semantics, configuration tips, and performance optimizations for building reliable high‑throughput data pipelines.
KafkaPerformance OptimizationReplication
0 likes · 19 min read

This article explains the design, semantics, and practical usage of Apache Kafka's transaction API, covering why transactions are needed for exactly‑once processing, the underlying atomic multi‑partition writes, zombie fencing, consumer guarantees, Java API details, performance considerations, and operational best practices.
Apache KafkaJavaTransactions
0 likes · 19 min read

This article explains the purpose, semantics, and design of Apache Kafka's transaction API, describes how exactly‑once processing is achieved in stream‑processing applications, outlines the Java client usage, and discusses the internal components, performance considerations, and best‑practice tips for developers.
JavaKafkaTransactions
0 likes · 16 min read

This article explains why exactly‑once semantics are needed for stream‑processing applications, describes Kafka's transactional model and semantics, details the Java transaction API and its usage, and discusses the internal components, performance trade‑offs, and practical guidelines for building reliable Kafka‑based pipelines.
JavaKafkaTransactions
0 likes · 17 min read

This article demonstrates how to achieve exactly‑once processing in Flink by providing Kafka I/O utility classes, a character‑count streaming example, and a transactional consumer implementation, while also discussing configuration nuances and common pitfalls.
FlinkJavaKafka
0 likes · 11 min read

This tutorial explains how to achieve exactly‑once semantics when streaming data from Kafka to Redis using Apache Flink's TwoPhaseCommitSinkFunction, covering Redis transaction basics, utility classes, sink implementation, testing steps, and solutions to common connection and transaction bugs.
FlinkJavaKafka
0 likes · 11 min read

Kafka is a high‑performance distributed message queue that uses topics, partitions, and replicas for scalability, with producers writing messages, consumers reading them via consumer groups, and advanced features such as offset management, delivery semantics, rebalancing, and exactly‑once transactional processing, all coordinated by Zookeeper.
ConsumersDistributed MessagingProducers
0 likes · 17 min read

This article provides a comprehensive overview of Kafka's architecture, covering topics, partitions, replication, producer and consumer workflows, offset management, rebalancing, delivery semantics, exactly‑once guarantees, transaction handling, file organization, and key configuration settings.
Distributed MessagingReplicationconsumer groups
0 likes · 17 min read

This article explains the technical details of how a serverless function compute platform provides exactly‑once task triggering semantics, covering message delivery models, system‑side failover handling, user‑side deduplication with TaskID, and a Go SDK example for reliable asynchronous job execution.
cloud-nativeexactly-oncetask deduplication
0 likes · 10 min read

Serverless Task provides exactly‑once task triggering by combining system‑side failover protection with user‑level deduplication, using a robust queue, scheduler, and globally unique TaskID to ensure each task is executed only once even under failures, network partitions, or retries.
Function Computecloud computingexactly-once
0 likes · 10 min read

This article explains how Apache Flink achieves end‑to‑end exactly‑once semantics by using source replay support, checkpoint‑based snapshots, asynchronous incremental checkpoints, and two‑phase commit sinks, and describes the interaction with external systems such as Kafka to ensure transactional writes.
FlinkKafkaTwo-Phase Commit
0 likes · 7 min read

RocketMQ-Streams targets massive data, high‑filtering, lightweight windowed computations with a lightweight, high‑performance design that runs on as little as 1 CPU core and 1 GB RAM, offering 2‑5× speed gains over traditional big‑data engines and supporting Flink‑compatible SQL, UDFs, and cloud‑native deployment.
Flink SQLRocketMQ-Streamsexactly-once
0 likes · 10 min read

This article explains the challenges of at‑least‑once delivery in distributed message middleware like RocketMQ, examines simple and concurrent deduplication techniques, and presents both transaction‑based and non‑transactional exactly‑once solutions using database tables or Redis, along with practical Java code examples.
RocketMQdeduplicationexactly-once
0 likes · 17 min read

This article explains the design goals, exactly‑once semantics, Java transaction API, coordinator and log architecture, and practical performance trade‑offs of Apache Kafka's transactional messaging, helping developers build reliable stream‑processing applications.
JavaKafkaTransactions
0 likes · 16 min read

This article explains why RocketMQ guarantees at‑least‑once delivery, describes the three typical duplicate‑message scenarios, compares transaction‑based and non‑transactional deduplication approaches (including a Redis‑based solution), provides sample SQL and Java code, and discusses the limitations and best‑practice recommendations for achieving idempotent message consumption.
IdempotencyJavaRocketMQ
0 likes · 20 min read

This article examines the true meaning of consistency in stream computing, clarifies common misconceptions about exactly‑once semantics, formalizes consistency challenges, and reviews how major stream engines such as Google MillWheel, Apache Flink, Kafka Streams, and Spark Streaming implement end‑to‑end consistency.
big dataexactly-oncefault tolerance
0 likes · 29 min read

Message middleware guarantees at-least-once delivery, but duplicate deliveries can cause issues; this article explains RocketMQ’s three duplication scenarios, explores simple and advanced deduplication strategies—including database-transaction and non-transactional approaches using Redis—and provides practical code samples for implementing reliable exactly-once processing.
RocketMQdeduplicationdistributed-systems
0 likes · 21 min read

This article explains why message middleware guarantees at‑least‑once delivery, the problems caused by duplicate consumption, and presents both transaction‑based and non‑transactional deduplication solutions—including a Redis‑backed idempotent table—to achieve exactly‑once semantics in RocketMQ.
IdempotenceRocketMQdistributed-systems
0 likes · 19 min read

This article analyzes the at‑least‑once guarantee of message middleware, explains why duplicate deliveries occur, compares simple database‑based deduplication methods, explores concurrency challenges, and presents a non‑transactional, status‑driven idempotence solution using MySQL or Redis with practical code examples and limitations.
IdempotenceMySQLRocketMQ
0 likes · 19 min read

This article explains why message middleware guarantees at‑least‑once delivery, describes three common duplication scenarios in RocketMQ, and presents both transactional and non‑transactional deduplication solutions—including SQL examples and a Redis‑based idempotence library—to achieve exactly‑once processing.
IdempotenceRocketMQdistributed-systems
0 likes · 19 min read

The article compares how RocketMQ, Apache Kafka, and Apache Pulsar implement transactional messaging—each using a two‑phase commit with half‑messages or transaction buffers, distinct coordinators, and idempotent producers—to provide atomicity and either strong consistency (RocketMQ) or exactly‑once delivery for high‑throughput stream processing (Kafka and Pulsar).
KafkaPulsarRocketMQ
0 likes · 22 min read

Message middleware guarantees at-least-once delivery, but repeated deliveries cause duplicate processing; this article explains RocketMQ's duplicate scenarios, explores simple and advanced deduplication techniques—including database-transactional and Redis-based idempotent tables—and provides practical Java code for implementing exactly-once semantics.
IdempotenceRocketMQdistributed-systems
0 likes · 20 min read

This article explains why message middleware guarantees at‑least‑once delivery, the challenges of duplicate consumption, and presents both transactional and non‑transactional deduplication strategies—including database and Redis‑based solutions—to achieve idempotent processing in RocketMQ.
IdempotenceRocketMQdeduplication
0 likes · 19 min read

This article explains why message middleware guarantees at‑least‑once delivery, the challenges of duplicate consumption, and presents both simple and advanced deduplication strategies—including transactional and non‑transactional approaches using relational databases or Redis—to achieve effectively exactly‑once semantics in RocketMQ.
IdempotenceRocketMQTransactional Messaging
0 likes · 18 min read

This article explains the core concepts of message queues—including producers, consumers, messages, brokers, and push/pull delivery—and analyzes three consumption scenarios: at‑most‑once, at‑least‑once, and exactly‑once, detailing the required producer, broker, and consumer behaviors for each.
ConsumerMessage Brokerexactly-once
0 likes · 6 min read

This article explains why Kafka is used, its buffering, decoupling, redundancy and robustness benefits, details the ack reliability levels, storage design, replica synchronization, ISR handling, consumer partition assignment strategies, transaction support, exactly‑once semantics, and why read‑write separation is not provided.
Consumer AssignmentReplicationexactly-once
0 likes · 20 min read

Apache Pulsar 2.8.0 introduces transaction support, featuring a Transaction Coordinator, Transaction Buffer, Transaction Log, Transaction ID, and Pending Acknowledge State, with a detailed workflow that ensures exactly‑once semantics for stream processing, contrasting its design with Kafka’s approach.
Apache PulsarKafka ComparisonTransactions
0 likes · 13 min read

This article explains Kafka’s three delivery semantics—At most once, At least once, and Exactly once—from both producer and consumer perspectives, details the required configuration settings, and discusses how Kafka ensures idempotence, transaction support, and prevents data loss, duplication, and ordering issues.
Delivery GuaranteesIdempotenceKafka
0 likes · 23 min read

This article explains why message middleware like RocketMQ guarantees at‑least‑once delivery, the resulting duplicate‑delivery problem, and presents both transaction‑based and non‑transactional idempotency solutions—including select‑for‑update, optimistic locking, and a Redis‑backed deduplication table—to achieve exactly‑once semantics in distributed systems.
Message IdempotencyRocketMQdeduplication
0 likes · 16 min read

This article explains Kafka's exactly‑once semantics and transaction support, detailing the new producer API methods, related exceptions, configuration parameters, and a sample application illustrating how to initialize, begin, process, and commit or abort transactions while ensuring idempotent and atomic message handling.
ConfigurationIdempotenceKafka
0 likes · 19 min read

This article explains common scenarios where Kafka messages can be lost on the producer, consumer, or broker side, and provides practical configurations—including callbacks, acks, retries, manual offset commits, idempotent and transactional producers—to ensure reliable delivery and exactly‑once processing.
IdempotenceMessage LossTransactions
0 likes · 10 min read

This article explains the three message delivery semantics in distributed stream processing, compares Kafka‑Spark Streaming integration methods (receiver vs direct stream), and details how to achieve exactly‑once guarantees through idempotent or transactional writes, including code examples.
@TransactionalKafkaSpark Streaming
0 likes · 8 min read

The article thoroughly explains Apache Flink’s checkpoint mechanism—including state types, coordinator workflow, exactly‑once versus at‑least‑once semantics, common failure sources such as code exceptions, storage or network issues, and practical configuration tips like interval settings, local recovery and externalized checkpoints.
Apache FlinkReal-time ComputingState Management
0 likes · 15 min read

This article explains how to develop a real‑time ETL application using Apache Flink that reads events from Kafka, partitions them by event time into HDFS directories, and achieves exactly‑once processing through checkpointing, custom bucket assigners, and proper state backend configuration.
Apache FlinkHDFSKafka
0 likes · 11 min read

This article explains the design goals, exactly‑once semantics, Java transaction API, internal components such as the transaction coordinator and log, data‑flow interactions, performance considerations, and practical tips for using Apache Kafka transactions in stream‑processing applications.
JavaKafkaTransactions
0 likes · 15 min read

This article describes how 58.com’s commercial engineering team redesigned its real‑time feature‑mining pipeline—replacing a minute‑level Spark Streaming framework with Flink—to achieve sub‑second latency, higher throughput, stronger fault‑tolerance, and end‑to‑end exactly‑once semantics for user‑profile generation in the second‑hand‑car recommendation scenario.
FlinkSparkbig data
0 likes · 14 min read

This article demonstrates how to develop a real‑time ETL job using Apache Flink, covering project setup, Kafka as a source, custom bucket assigners for HDFS, checkpointing, savepoints, and deployment on YARN to achieve exactly‑once processing guarantees.
Apache FlinkKafkaStreamingFileSink
0 likes · 11 min read

This article explores the two toughest problems in distributed stream processing—exactly‑once message handling and ordering—by dissecting the underlying impossibility of perfect failure detectors, the liveness‑vs‑safety trade‑off, zombie processes, and the practical solutions employed by systems such as Flink, Kafka Streams, MillWheel, and Spark.
Consensuscheckpointingdistributed-systems
0 likes · 81 min read

This article explains Kafka's Exactly-Once Semantics (EOS), the role of idempotence, and how transactional support works, covering EOS semantics, producer id and sequence numbers, configuration properties, and providing Java code examples for initializing, beginning, committing, and aborting transactions.
IdempotenceJavaKafka
0 likes · 8 min read

This article provides a comprehensive overview of Kafka's architecture, covering topics, partitions, producer and consumer workflows, replication and leader election, offset management, consumer group coordination, rebalance processes, delivery semantics (at‑most‑once, at‑least‑once, exactly‑once), transactional messaging, and underlying file and configuration details.
Distributed MessagingKafkaReplication
0 likes · 16 min read

This article compares Apache Spark and Apache Flink, outlining their programming models, streaming mechanisms, state management, time semantics, and exactly‑once guarantees, and highlights the strengths and differences of each framework for batch and real‑time big‑data processing.
Apache FlinkApache SparkState Management
0 likes · 8 min read


ITPUBMar 28, 2019 · Big Data
Pravega, Dell’s native stream storage project, addresses the challenges of modern low‑latency, exactly‑once stream processing by combining tiered storage, Apache BookKeeper, and seamless Flink integration, offering a unified solution that reduces development, storage, and operational costs compared to traditional message systems like Kafka.
Apache FlinkKafka ComparisonPravega
0 likes · 10 min read

This article explains Apache Flink's fault‑tolerance mechanisms, including checkpointing, barrier alignment, the differences between At‑Least‑Once and Exactly‑Once semantics, configuration options, incremental checkpointing, and the requirements for external sources and sinks to achieve end‑to‑end exactly‑once processing.
Apache Flinkbig datacheckpointing
0 likes · 15 min read

This article analyzes the difficulties of achieving exactly-once delivery in Apache Flink, explains the distinction between state and end‑to‑end semantics, and details how idempotent and transactional sinks—illustrated with the Bucketing File Sink—realize exactly‑once guarantees through checkpoint‑based two‑phase commit.
FlinkState ManagementTransactional Sink
0 likes · 13 min read

This article explains how network unreliability makes message delivery in distributed systems complex, covering network request outcomes, timeout handling, the three delivery semantics—at‑most‑once, at‑least‑once, exactly‑once—and how protocols like AMQP and MQTT address these challenges.
AMQPAt-Least-OnceMQTT
0 likes · 15 min read


21CTOSep 5, 2018 · Fundamentals
The article explains how messages travel in monolithic and distributed systems, why network communication is inherently unreliable, and how different delivery semantics—at‑most‑once, at‑least‑once, and exactly‑once—affect reliability, ordering, and protocol choices such as AMQP and MQTT.
At-Least-OnceMessage Deliverydistributed-systems
0 likes · 14 min read

This article provides a comprehensive overview of Apache Kafka, comparing it with ActiveMQ, explaining its distributed architecture, topics, partitions, consumption models, high‑availability mechanisms, exactly‑once semantics, and includes detailed Java producer and consumer code examples for practical implementation.
ConsumerDistributed MessagingKafka
0 likes · 22 min read

This article compares Spark Streaming and Flink across runtime models, component roles, programming APIs, task scheduling, time semantics, dynamic Kafka partition detection, fault‑tolerance mechanisms, exactly‑once guarantees, and back‑pressure handling, providing code examples and practical insights for real‑time data processing.
Dynamic Partition DetectionFlinkKafka Integration
0 likes · 23 min read

This article details Lianjia's journey of designing and implementing a low‑latency, stable real‑time computing platform using Spark Streaming on YARN, covering technical selection, architecture components, version compatibility challenges, exactly‑once semantics, graceful shutdown, Kafka tuning, and future enhancements.
KafkaSpark StreamingYARN
0 likes · 11 min read

This article explores how distributed systems determine node liveness, manage failover and recovery, and implement at‑most‑once, at‑least‑once, and exactly‑once processing guarantees—including opaque transactions and two‑phase commit—using examples from Kafka, Zookeeper, and big‑data pipelines.
FailoverZooKeeperbig data
0 likes · 15 min read

This article presents a comprehensive performance evaluation of Apache Flink versus Apache Storm across multiple real‑time processing scenarios, measuring throughput, latency, and the impact of different configurations and delivery semantics to guide framework selection and optimization.
FlinkStormexactly-once
0 likes · 16 min read

The study benchmarks Apache Flink against Apache Storm on a shared cluster, showing Flink delivering three‑to‑five times higher throughput and roughly half the latency across simple, sleep‑induced, and windowed workloads, with modest throughput loss for exactly‑once semantics, leading to a recommendation of Flink for high‑performance, stateful real‑time stream processing.
Apache FlinkApache StormReal-time Stream Processing
0 likes · 19 min read

This article describes how Ctrip migrated its large‑scale real‑time platform from JStorm to Spark Streaming, detailing the architectural design, the Muise Spark Core encapsulation, operational metrics, encountered pitfalls, and future plans to adopt Flink and Beam for streaming workloads.
Spark StreamingYARNbig data
0 likes · 22 min read

This article explores practical techniques for handling node liveness, failover, recovery, and exactly‑once transaction semantics in distributed systems, illustrating implementations with Zookeeper, Kafka, Storm, and database sharding while addressing big‑data reach calculations and performance trade‑offs.
Transactionsbig datadistributed-systems
0 likes · 15 min read
