Flink Forward 2017: Stream Processing Insights from Alibaba, Uber & Netflix

In April 2017 the third Flink Forward conference was held in San Francisco, gathering engineers from Google, Uber, Airbnb, Amazon, Apple, Facebook and other leading tech companies.

Wang Shaoxuan (nickname "Da Sha"), an Alibaba engineer who previously worked on Facebook's TAO distributed graph database, attended and shared his impressions.

Apache Flink originated from a research project at Berlin University of Technology, initially called Stratosphere, open‑sourced in 2014 and later commercialized by dataArtisans.

Stephan Ewen, CTO of dataArtisans, gave two talks: a high‑level overview of real‑time analytics and a deep dive into large‑scale Flink deployments.

Key use cases presented included:

ING using Flink to detect credit‑card fraud in real time.

Uber’s AthenaX platform, built on Flink SQL, for real‑time analytics, machine‑learning data pipelines and anti‑fraud detection.

Netflix’s Keystone pipeline, a multi‑tenant, fault‑tolerant streaming‑as‑a‑service on AWS, processing up to one trillion events per day.

Alibaba’s Blink, an internal real‑time engine based on Flink, supporting billions of events per second across dozens of core business lines.

Stephan emphasized the evolution of stream processing from batch‑oriented “lambda” architectures to native streaming frameworks that provide low latency and exactly‑once guarantees.

He also highlighted future directions such as event‑driven applications and the integration of Flink SQL into Alibaba’s real‑time machine‑learning platform.

Additional sessions covered detailed runtime improvements for large‑scale streaming, checkpointing, state back‑ends, and the development of Flink TableAPI/SQL, to which Alibaba contributed heavily.

Alibaba’s Blink now runs on over 1500 machines, handling billions of events per second and petabytes of state, and was used successfully during the 2017 Double 11 shopping festival.

The conference demonstrated growing industry adoption of Flink, with more than 90 companies presenting, and underscored Alibaba’s role as the largest contributor to the Flink project.

Finally, the author invited engineers interested in real‑time stream processing, SQL, and TableAPI to contact him, and provided links to session videos, slides, and related articles.