Summary of Flink Forward Asia 2021: Community Growth, Cloud‑Native Deployment, Streaming‑Batch Integration, and Machine Learning

Flink Forward Asia (FFA) 2021, an online summit authorized by Apache and hosted by the Apache Flink Chinese community, gathered developers and users to review a year of progress and showcase upcoming directions.

The community report highlighted that Flink has been one of the most active Apache projects for three consecutive years, with over 1,400 contributors and a 20% annual growth rate. The Chinese community alone has more than 50,000 WeChat followers, over 140 technical articles, a dedicated video channel, and a revamped learning site (Flink Learning) that aggregates tutorials, case studies, and event information.

From an industry impact perspective, Flink is now the de‑facto standard for real‑time computing. Over 100 companies contribute code, and major sponsors such as Alibaba, ByteDance, Ctrip, and 360 support meet‑ups. The conference featured 40+ companies from internet, finance, energy, manufacturing, and telecom sectors delivering 83 keynote talks.

Cloud‑Native Deployment – Flink’s deployment evolved from a static Standalone mode to an Active mode tightly integrated with Kubernetes/YARN, and finally to an Adaptive/Reactive mode that automatically scales resources without user intervention, lowering operational complexity.

Fault Tolerance – Checkpointing – Checkpointing remains central to Flink’s resilience. Two major projects aim to improve reliability and latency: Buffer Debloating (reducing upstream buffering) and Generalized Log‑Based Checkpoint (decoupling snapshot creation from asynchronous state upload), expected in Flink 1.15.

Streaming‑Batch Integration – The “Streaming Warehouse” concept unifies batch and stream processing at both compute and storage layers. API‑level unification introduced a unified SQL/Table API and an Imperative DataStream API, while the connector framework now supports both streaming and batch sources. Alibaba’s Remote Shuffle Service and the Flink‑extended project provide a pluggable shuffle layer.

Two flagship use cases illustrate the Streaming Warehouse:

Flink CDC enables full‑incremental data integration by first syncing historical data and then continuously streaming binlog changes.

Streaming Warehouse (StreamHouse) combines Dynamic Tables (which store data in both LSM‑Tree and log formats) with Flink SQL to achieve real‑time OLAP on the same data pipeline.

Machine Learning – Flink ML 2.0 – Built on the unified DataStream API, Flink ML 2.0 offers a Scikit‑Learn‑style pipeline, native iterative computation, and integration with PyTorch/TensorFlow, enabling real‑time and batch ML workloads. PyFlink now achieves near‑Java UDF performance through a JNI‑based runtime.

ByteDance Experience – ByteDance migrated all streaming jobs from JStorm to Flink by 2019, now runs over 100,000 Flink jobs daily (30% SQL), consumes >4 million CPU cores, and peaks at 90 billion QPS with 600 GB/s checkpoint traffic. Their BMQ message system, built on a storage‑compute‑separated architecture, provides tiered storage and auto‑scaling for both stream and batch consumption.

Industrial Application – ICBC – The Industrial and Commercial Bank of China uses Flink for real‑time ETL, data lake ingestion, and downstream services (HBase, Elasticsearch, Presto). Dynamic Tables enable a single Flink pipeline to handle ingestion, serving, and BI analytics, while data security is enforced through lifecycle monitoring, watermarking, row‑level access control, and ML‑based sensitive data detection.

Pravega – Deconstructing Stream Storage – Pravega offers a unified stream‑batch storage system with ordered writes, transactional guarantees, checkpointing, and tiered storage built on distributed object stores. Its auto‑scaling and edge‑computing optimizations complement Flink’s reactive scaling, and joint roadmap work aims to push computation to the data source.

Round‑Table Discussions – Leaders from Alibaba, ByteDance, Meituan, Kuaishou, Xiaomi, ICBC, Dell, and Xiaohongshu debated topics such as the maturity of Flink, future directions for real‑time analytics, open‑source community sustainability, and the balance between innovation and production stability.

Conclusion – 2021 marked a turning point for big‑data streaming, with Flink’s SQL maturing, streaming‑batch integration gaining industry acceptance, and the emergence of Streaming Warehouse as a strategic focus. The community aims to continue advancing unified APIs, storage, and AI integration to realize truly end‑to‑end real‑time data pipelines.