Key Takeaways from Flink Forward 2021: Real‑Time Computing, Flink SQL, ML, and Streaming Warehouse

Real‑time Computing Becomes Common Across Traditional Industries

Two years ago real‑time computing seemed impossible, but in the past two years it has exploded thanks to strong community efforts, especially from the Flink community.

Alibaba has invested heavily, with a large operations team and mature cloud products, leading banks, insurance, securities and other non‑internet companies to explore Flink‑based real‑time systems.

Real‑time computing is now a core skill for data professionals; those focused on traditional data warehouses need to shift toward real‑time architectures.

Flink SQLization Is Inevitable

The data development field is moving toward SQL everywhere, which benefits businesses but may pose challenges for developers who fear automation of their work.

Many companies are migrating their compute platforms to SQL, offering great convenience for business developers.

Flink ML

The community has designed native APIs for real‑time machine learning, supporting flexible configuration, composition, and deployment of online prediction and learning algorithms, with multi‑input/multi‑output support and a graph‑based module composition.

They plan to integrate Alibaba Cloud’s Alink algorithm library into Flink ML, combining Flink’s ecosystem with Alink’s capabilities to create a comprehensive streaming‑batch machine‑learning library.

This effort aims to close Flink’s ML gap compared with Spark and could lead to new commercial offerings.

Streaming Warehouse (Streamhouse) – A New Force

Flink’s community leader Wang Feng outlined the next step for Flink: moving from stream processing to a Streaming Warehouse that unifies real‑time and batch analytics.

The vision is to capture data changes at the source, enable layer‑by‑layer real‑time analysis, and support both real‑time and offline queries with a single API.

Flink Dynamic Table, discussed in FLIP‑188, provides a unified storage that integrates seamlessly with Flink SQL, allowing creation of dynamic tables that can be queried in real time or processed in batch.

The first implementation of Dynamic Table is complete, and future work aims to service‑ify it, potentially turning it into an independent project for general streaming‑batch storage.

Combining Flink CDC, Flink SQL, and Dynamic Table can deliver a full streaming‑batch data warehouse experience.

Finally, the author encourages readers to keep learning and stay engaged.