Why 2020 Was the Breakthrough Year for Apache Flink’s Ecosystem

2020: A Year of Rapid Growth for Apache Flink

Apache Flink became the most active Apache project in 2020, topping mailing‑list activity, GitHub commits, and visitor metrics. The community saw average annual growth of over 30% in stars and contributors, indicating a healthy and fast‑moving ecosystem.

Annual Release Highlights

Three major releases—Flink 1.10, 1.11, and 1.12—delivered significant advances:

Unified stream‑batch architecture with SQL support for both modes.

CDC integration for reading database binlogs.

Extensive Python support (PyFlink) enabling full‑stack development.

Native Kubernetes deployment, removing Hadoop dependencies.

Chinese Community Expansion

The Flink Chinese community launched a dedicated mailing list, a WeChat public account with over 30,000 subscribers, and a learning site (https://flink-learning.org.cn/), providing weekly updates and best‑practice articles.

Flink as the Real‑Time Computing Standard

Flink is now the de‑facto standard for real‑time analytics, adopted by dozens of leading companies worldwide and featured at Flink Forward Asia 2020.

Technical Innovations

Unaligned Checkpoint – Enables faster checkpoints under back‑pressure by switching to a non‑aligned mode.

Approximate Failover – Provides a more flexible fault‑tolerance model that restarts only the failed node, avoiding full DAG restarts.

Nexmark Benchmark – A streaming benchmark suite with 16 SQL queries covering common stream‑processing scenarios.

Architecture Evolution

The new unified architecture supports both bounded (batch) and unbounded (stream) data, with Table/SQL and DataStream APIs offering consistent semantics. Runtime improvements include pluggable scheduling and a Remote Shuffle Service that can run on Kubernetes.

Performance Benchmarks

TPC‑DS tests show Flink 1.12 achieving three‑fold speedup over Flink 1.9, completing a 10 TB workload on 20 machines in under 10 000 seconds, matching leading batch engines.

Stream‑Batch Integration for Data Integration

Flink SQL now supports CDC‑based synchronization from databases to data warehouses (Hive, ClickHouse, TiDB) and can seamlessly mix batch and stream operators.

Data‑lake integrations include Flink + Iceberg and Flink + Hudi, offering ACID guarantees, snapshotting, and upsert capabilities.

AI Integration

PyFlink matured with full Table and DataStream APIs, Python UDF/UDTF/UDAF support, and Pandas integration. The Alink library added dozens of machine‑learning algorithms, and the Flink AI Flow project combines deep‑learning frameworks (TensorFlow, PyTorch) with Flink for real‑time ML pipelines.

Native Kubernetes Deployment

Flink 1.12 supports native K8s deployment, including HA, dynamic resource scaling, and GPU/CPU scheduling, making it ready for cloud‑native environments.

Flink at Alibaba

Since 2016, Alibaba has scaled Flink across the Double 11 shopping festival, achieving full‑chain real‑time data processing and a one‑year resource‑doubling without additional hardware. The company now runs Flink‑based real‑time and batch workloads together, leveraging Hologres for unified storage and achieving 4‑10× faster report development.

Looking ahead, Alibaba and the broader community aim to push stream‑batch unification, real‑time‑offline integration, and big‑data‑AI convergence forward.