Understanding Flink’s Exactly-Once Semantics and Its Relation to Deduplication

Flink’s Exactly‑Once semantics is a common interview question, especially after some practitioners mistakenly assumed it guarantees true once‑only processing and suffered interview failures.

The issue splits into two parts: what does “Exactly‑Once” mean in Flink, and how to achieve truly exactly‑once computation.

In production, relying solely on Flink’s built‑in Exactly‑Once guarantee is difficult, particularly for strict scenarios such as precise order‑volume statistics.

Since Flink 1.4, the TwoPhaseCommitSinkFunction provides a generic two‑phase commit implementation, making end‑to‑end Exactly‑Once possible when paired with compatible sources and sinks (e.g., Kafka 0.11).

However, Exactly‑Once does not mean each event is processed only once; engines cannot guarantee a user’s logic runs exactly once per event under failures, and duplicate processing may occur.

What Flink guarantees is that the engine‑managed state updates are committed exactly once; achieving end‑to‑end Exactly‑Once requires source and sink support as well.

Flink’s documentation notes that offsets are not used for fault‑tolerance but for monitoring, and when checkpointing is enabled the Kafka consumer commits offsets after each successful checkpoint, which can be disabled via setCommitOffsetsOnCheckpoints(false).

Thus, Flink’s Exactly‑Once relies on checkpointing to ensure state consistency.

To realize true exactly‑once computation, the source and sink must be immune to duplicate data, which typically involves three safeguards: using external storage to guarantee unique delivery, implementing deduplication (e.g., sorting, distinct) in Flink jobs with checkpoints enabled, and ensuring the sink is idempotent.

When these three guarantees are in place, the checkpoint mode’s Exactly‑Once setting becomes less critical, and the two questions should not be conflated.