How Flink’s Streaming Warehouse Is Redefining Real‑Time Data Lakes

Flink’s Evolving Role in Big Data

Historically, Apache Flink did not provide its own storage system but offered connectors to services such as Amazon Kinesis, Apache Kafka, Alluxio, HDFS, Apache Cassandra, and Elasticsearch. This limitation is now being addressed as Flink moves toward a unified streaming‑batch architecture.

Streaming‑Batch Integration: Two Use Cases

At Flink Forward Asia 2021, Wang Feng (aka “Mo Wen”) highlighted two key scenarios enabled by Flink’s stream‑batch integration.

1. Incremental data integration with Flink CDC. By writing a single SQL statement, users can perform a full historical data sync followed by automatic incremental updates, eliminating the need for separate offline and real‑time pipelines.

Flink CDC Connectors, now at version 2.1, support major databases such as MySQL, PostgreSQL, MongoDB, Oracle, and are expanding to TiDB and DB2.

2. Real‑time data warehouse (Streaming Warehouse). Traditional architectures combine Flink + Kafka for real‑time processing and a separate offline warehouse, leading to duplicated APIs, inconsistent data semantics, and complex data pipelines. The Streaming Warehouse concept proposes a unified API that enables end‑to‑end real‑time data flow, consistent data semantics, and seamless offline analytics.

Key Technical Advances in Flink 1.14

Flink 1.14 introduces mixed bounded and unbounded stream support, checkpointing at the end of bounded streams, unified Source and Sink APIs, and hybrid sources that can transition between storage systems (e.g., from Amazon S3 to Apache Kafka). Batch execution now allows mixing DataStream API with SQL/Table API.

Understanding the Streaming Warehouse

The Streaming Warehouse aims to make the entire data warehouse “streaming” by moving data continuously through layered storage using a single set of APIs. It supports real‑time queries, incremental updates, and batch ETL, while maintaining data consistency and simplifying architecture.

Dynamic Table Storage: Flink Dynamic Table

To realize the Streaming Warehouse, Flink proposes Dynamic Table storage, comprising a File Store (LSM‑based, columnar, supporting batch reads) and a Log Store (immutable log for streaming reads). This storage integrates seamlessly with Flink SQL, enabling both real‑time and batch operations on the same tables.

Flink Beyond Computation

Flink is expanding from a pure stream processor to a platform that includes stateful storage, unified APIs, and integration with external systems, positioning it to address end‑to‑end real‑time analytics challenges.

Conclusion

The industry is moving toward integrated, simplified data architectures. Flink’s Streaming Warehouse and Dynamic Table initiatives represent early steps in this direction, with a preview expected in Flink 1.15 and broader adoption anticipated as the community matures the solution.