Implementing End-to-End Exactly-Once Semantics in Apache Flink with Apache Kafka Using Two-Phase Commit Sink
This article explains how Apache Flink’s TwoPhaseCommitSinkFunction, introduced in version 1.4, enables end-to-end exactly-once semantics when integrated with Apache Kafka, detailing the checkpoint mechanism and the two-phase commit protocol that ensures reliable data processing.
The following content is based on a presentation by an Alibaba engineer at the Apache Kafka × Apache Flink conference in Beijing, discussing the principles of achieving end-to-end consistency semantics by combining Apache Flink with Apache Kafka.
In December 2017, the Apache Flink community released version 1.4, which introduced a milestone feature: the Two‑Phase Commit sink, implemented as the abstract class TwoPhaseCommitSinkFunction.
This sink function encapsulates the common logic of the two‑phase commit protocol, making it possible for Flink to build exactly‑once processing applications when paired with appropriate sources and sinks. Users extend the abstract class to implement the required methods for exactly‑once semantics.
The presentation then covers two key aspects: (1) the role of Flink checkpoints in guaranteeing exactly‑once semantics, and (2) how Flink uses the two‑phase commit protocol to ensure exactly‑once semantics from data source to data sink.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
Big Data Technology & Architecture
Wang Zhiwu, a big data expert, dedicated to sharing big data technology.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.