Pravega Flink Connector: Past, Present, and Future – Architecture, Checkpoint Integration, and Upcoming Features

Pravega is an open‑source, CNCF sandbox project created in 2016 for large‑scale data‑stream scenarios, offering high‑performance, elastic, tiered storage that addresses the limitations of traditional message queues.

The Pravega Flink connector, first released as an independent GitHub project in 2017, initially provided basic source and sink functions and later added exact‑once semantics through a two‑phase commit mechanism, eventually supporting batch reads and Table API integration, and focusing on Flink 1.11 features such as FLIP‑27 and FLIP‑95.

Unlike Kafka, Pravega implements its own checkpoint mechanism: the Job master triggers checkpoints via the ExternallyInducedSource interface, Pravega uses a StateSynchronizer to broadcast checkpoint events to all readers, and completed checkpoints are stored in the Job master state, avoiding tight coupling with Flink internals.

The integration of Flink 1.11 presented challenges, exemplified by issue FLINK‑18641, which required debugging of the CheckpointCoordinator's mailbox model; the experience highlighted the importance of searching mailing lists, providing detailed logs, and collaborating with the community.

Future work includes integrating the Pravega schema‑registry to simplify Table API usage, adopting upcoming Flink features (FLIP‑143, FLIP‑129), and supporting a Docker‑based test framework, while also promoting the Pravega maker competition to encourage community involvement.