What I Learned at Flink Forward Asia 2019: Stream Processing, AI, and Cloud‑Native Insights

From November 28‑30, 2019, Beijing hosted the first winter snow and the Flink Forward Asia (FFA) conference, drawing over 2,000 participants—almost double the previous year.

FFA is an Apache‑authorized event held annually on three continents, offering the latest Flink community updates, development roadmaps, and production experiences across a diverse set of more than 25 companies, universities, and research institutes.

Main Session Topics

Stateful Functions

Stephan Ewen presented the vision of Flink as a general engine for application services, emphasizing the need for integrated batch, streaming, and online processing. He explained that traditional FaaS solutions suffer from state access bottlenecks, consistency problems, scalability limits, and connection overhead, which Flink’s stream processing can address. Stateful Functions extend Flink with function composition and lightweight virtual instances, providing clear abstraction by separating message transport, state management, and computation logic.

Unified Engine & AI

Wang Feng (Alibaba) outlined Flink’s progress toward a unified engine that combines batch, streaming, and application workloads, while also embracing AI and cloud‑native environments. Flink 1.9/1.10 unified SQL/Table API for batch and streaming, added full DDL support, Hive compatibility, and Python UDFs.

Benchmarks showed Flink 1.10 outperforming Hive 3.0 by a factor of seven on TPC‑DS. The AI focus introduced a standardized Machine Learning Pipeline API (inspired by Scikit‑learn) and better Python support, while Alibaba’s Alink library brought streaming‑enabled machine learning to Flink.

AI Flow was highlighted as a forthcoming project to provide an end‑to‑end solution for data acquisition, preprocessing, model training, validation, serving, and inference.

Cloud‑Native Integration

Deep integration with Kubernetes was discussed, enabling multi‑user isolation, improved stability, and native resource management. Flink 1.10 can run natively on Kubernetes, and upcoming talks addressed pluggable shuffle services, unaligned checkpoints, and YuniKorn scheduling.

Sub‑Session Tracks

Apache Flink Core Technology – deep dives into 1.9/1.10 updates.

Artificial Intelligence – Flink + TensorFlow use cases.

Enterprise Practice – experiences from ByteDance, Kuaishou, Didi, NetEase, iQIYI, Bilibili, 360, etc.

Real‑time Data Warehouse – implementations at Netflix, Meituan, Xiaomi.

Open‑Source Big Data Ecosystem – integrations with Zeppelin, Hive, Pulsar, Pravega, etc.

Highlights and Demos

Key demos included a Flink + Hive + Zeppelin SQL workflow, Alink ML in Jupyter, and Lyft’s large‑scale near‑real‑time analytics platform that injects streaming data via Flink, persists to S3 in Parquet, and performs multi‑stage non‑blocking ETL for an interactive data warehouse.

Lyft reported a 10% reduction in cluster size for data ingestion compared to a Kinesis‑based solution and discussed challenges around checkpointing, sub‑task stalls, and the need for robust Kubernetes deployment.

Pravega was presented as a streaming storage system offering exactly‑once semantics and scalable stream segments, contrasting with traditional message queues.

Conclusion

The conference reinforced Flink’s trajectory toward engine integration, ecosystem diversification, and cloud‑native adoption. The author reflects on the community’s rapid growth and calls for continued collaboration to sustain Flink’s momentum into 2020 and beyond.