Why Apache Flink Became the Fastest‑Growing Big Data Engine in 2018

Recent market research shows Apache Flink was the fastest‑growing open‑source big data engine in 2018, increasing 125% over 2017. To help readers understand Flink comprehensively, a digital collection titled Not Just Stream Computing: Apache Flink Practice compiles large‑scale implementations from top Chinese internet companies.

In the collection you can learn how Flink helped Alibaba smoothly handle Double‑11, how Didi built simple APIs for its complex business needs, and how ByteDance replaced JStorm with Flink as its sole streaming engine.

Flink is recognized as the best stream processing engine, but its capabilities extend to batch, interactive analytics, and machine learning, making it a versatile big‑data engine for AI computing.

Big‑data computing typically falls into three categories:

Batch processing – large data volumes with relaxed latency.

Stream processing – low‑latency, fixed queries executed as data arrives.

Interactive analytics – ad‑hoc queries requiring near‑real‑time responses.

Artificial intelligence workloads often combine batch and stream processing in an ML pipeline that includes data preprocessing, feature extraction, model training, and validation, with interactive analysis essential for data exploration.

Alibaba chose Apache Flink as a unified engine to address these challenges, leveraging its low latency, high throughput, exactly‑once guarantees, and strong performance in both batch and streaming scenarios, as well as its suitability for IoT and AI workloads.

Flink’s community and ecosystem continue to grow, with improvements in Table API, Python, ML integration, and compatibility with Hive, Zeppelin, and Jupyter, aiming to provide a single engine for all big‑data intelligent computing needs.

To bring Flink closer to Chinese developers, Alibaba hosted the first Flink Forward China conference in Beijing, gathering over 1,000 developers, architects, data scientists, and core contributors to share best practices from Alibaba, Didi, Meituan, ByteDance, Uber, and others.