How Apache Flink Powers Real‑Time Big Data at Alibaba and Beyond

Flink Forward China 2018, organized by Alibaba Group, took place at the Beijing National Convention Center in late December 2018. It was the first time the Apache‑sponsored global Flink conference was held in China.

The conference gathered speakers from Alibaba, Tencent, Huawei, Didi, Meituan‑Dianping, ByteDance, iQIYI, Qunar, Uber, DellEMC and the Flink founding company DA, who shared the history, application scenarios and future trends of Apache Flink.

Presentation slides and videos are available at https://github.com/flink-china/flink-forward-china-2018.

Apache Flink originated from the Stratosphere research project at Berlin University of Applied Sciences, was donated to the Apache Software Foundation in April 2014, and became a top‑level Apache project in December 2014. After four years of rapid growth, the Flink community now has 42 committers and 19 PMC members.

Alibaba has contributed over 230 open‑source projects, collaborates with more than 8,000 partners and 2,000 ISVs, and has invested more than 150,000 lines of code in Flink. Since 2015 Alibaba has been using Flink in search, and by 2018 Flink powers real‑time computation for all Alibaba subsidiaries, handling a peak of 1.7 billion events per second during Double 11.

In January 2019 Alibaba open‑sourced its internal Flink fork, Blink, to let users benefit early from Alibaba’s improvements. Blink also feeds back many enhancements to the upstream Flink project.

Key technical contributions from Alibaba to Flink Runtime include:

A new distributed system architecture that decouples job scheduling from resource management, enabling native execution on YARN and Kubernetes and allowing fully distributed job scheduling for larger clusters.

Enhanced fault‑tolerance with region‑based failover and job‑manager HA mechanisms, ensuring whole‑job restarts on any task or master failure.

Performance optimizations such as incremental checkpoints for TB‑scale state, async I/O to avoid back‑pressure from external storage, and a credit‑based network flow control.

In Flink SQL, Alibaba added streaming‑SQL semantics (Agg Retraction, UDX support, DDL extensions, many connectors) and improved the query execution and optimizer with better code generation and memory management.

Alibaba also pursued batch‑stream fusion: instead of maintaining separate Lambda‑style pipelines for real‑time and offline processing, they built a unified Flink‑based engine where a single code base serves both scenarios, simplifying development and resource usage.

Beyond Alibaba, the conference highlighted real‑time applications such as:

City‑level smart‑city analytics in Hangzhou, where Flink processes data from 1,300 traffic cameras and sensors to monitor traffic flow, predict congestion and even create green corridors for ambulances, saving up to 50 % of response time.

Industrial IoT monitoring for companies like GCL‑Poly, where real‑time device metrics improve product yield by 1 %.

Didi’s real‑time platform, which handles PB‑scale data, supports risk control, coupon distribution, anomaly detection and feature engineering via Flink DataStream, SQL and CEP APIs, and has contributed extensions back to the community.

The event demonstrated that Apache Flink has become the de‑facto streaming engine for major Chinese internet companies, with deployments ranging from thousands of nodes at Alibaba and Didi to over ten thousand nodes at ByteDance.

Future directions for Flink include deeper integration with machine‑learning and graph‑processing ecosystems, stronger event‑driven application support, and continued advances in batch‑stream fusion and AI‑centric workloads.