Is Apache Flink Truly Powerful Enough After Hundreds of Engineers and Multiple Double‑11 Deployments?

Apache Flink, one of the most active big‑data projects, became an Apache top‑level project eight years ago and now supports both streaming and batch execution with seamless Hadoop integration.

Alibaba began exploring Flink in 2015, launched it in a search scenario in 2016, and has since contributed over a million lines of code through the Blink fork. In the 2021 Double 11 event, Flink handled a peak of 4 billion records per second and 7 TB of data per second, illustrating its massive scale.

After beating Storm and Spark Streaming, Flink became the sole standard for stream computing, with no technical competitors.

Flink’s early advantage stemmed from its stateful stream processing model and distributed snapshot mechanism, which provide high‑performance pure streaming and strong consistency guarantees.

While Spark Streaming relies on the batch‑oriented Spark engine, limiting performance and expressiveness, Flink has matured its batch capabilities, passing the TPC‑DS benchmark and reaching performance comparable to mainstream batch engines.

The need for a unified stream‑batch approach arises from business shifts toward real‑time analytics such as risk control, BI, recommendation, and monitoring. Maintaining separate offline and online pipelines leads to duplicated development, inconsistent metrics, and higher operational complexity. A single engine that serves both reduces these issues and enables shared resource pools for search, recommendation, and advertising workloads.

However, the author cautions that stream‑batch integration is not universally beneficial; it makes sense when real‑time workloads dominate or when strict data consistency is required.

Stream‑batch Data Warehouse Architecture

Stream‑batch integration is a technical concept.

Flink’s SQL layer now offers unified semantics, allowing the same SQL to run on both streaming and batch data. Yet storage remains fragmented: streaming writes to Kafka, while batch writes to Hive, Iceberg, or Hudi, requiring separate maintenance.

Because no production‑ready storage currently supports efficient stream reads/writes and batch reads/writes simultaneously, the community is developing Flink Table Store. Built as an independent sub‑project in late 2021, Table Store aims to provide a truly unified storage layer for stream‑batch workloads.

In 2022, Table Store released two versions, with contributions from Alibaba, ByteDance, and other companies, and early adopters have begun testing it.

At Flink Forward Asia 2022, a full product demo showcased an end‑to‑end pipeline on Alibaba’s real‑time platform: CDC ingest from databases into Table Store, real‑time SQL analysis on Table Store, batch correction, and interactive queries, illustrating the complete stream‑batch data‑warehouse solution. The architecture also supports other storage systems such as Alibaba Cloud Hologres.

Full‑Incremental Data Integration

According to Gartner’s 2022 stream‑processing market guide, about one‑third of stream‑processing use cases involve real‑time data integration. Traditional stacks require separate tools for full‑load and incremental sync, making seamless, consistent integration difficult.

Leveraging Flink’s unified execution model, CDC achieves automatic mode switching, checkpoint‑based fault tolerance, and lock‑free incremental snapshot reads, enabling truly incremental, end‑to‑end data sync without impacting production workloads.

The Flink CDC project, now with over 3,000 GitHub stars, supports major databases (MySQL, Oracle, PostgreSQL, MongoDB, TiDB, PolarDB, OceanBase) and sees contributions from companies like NetEase, Tencent, and ByteDance.

Cloud‑Native Evolution

Flink has been designed for cloud‑native environments from the start, with native Kubernetes scheduling, stream‑shuffle, and state handling that can snapshot to HDFS or cloud storage. Deployments no longer depend on Hadoop; a Kubernetes cluster suffices.

Kubernetes brings isolation, multi‑tenant management, and paves the way for serverless Flink, where users can run jobs without managing underlying machines.

Alibaba Cloud offers a managed, serverless Flink product built on these cloud‑native capabilities, providing a SQL‑centric development and operations platform for real‑time warehousing, integration, risk control, and feature engineering, with plans to launch a multi‑cloud PaaS serverless service globally in the coming months.