What’s New in Flink? Insights from the 2018 Flink Forward Berlin Conference

Flink Forward is an international conference authorized by Apache and organized by dataArtisans, with participation from Alibaba, Uber, Airbnb, Netflix and others. The 2018 Berlin edition has just concluded.

Apache Flink is an open‑source distributed stream processing framework that provides high‑performance, high‑availability, exactly‑once guarantees, and supports both streaming and batch (via bounded streams) as well as SQL.

Leager: A New Take on ACID

At the conference dataArtisans announced Leager, a cloud‑native distributed‑transaction product that offers a new ACID solution with better performance than traditional distributed transactions. Two versions are available: a single‑node streaming version and a River version sold on the DA Platform. The source code is on GitHub.

Batch‑Stream Unification

Alibaba announced deep integration of batch and streaming models, achieving order‑of‑magnitude improvements in batch performance. Flink’s architecture, built on streaming, allows batch processing to be expressed as bounded streams, eliminating the gap between batch and stream.

Flink vs Spark

While Spark implements streaming on top of batch (RDD), Flink is natively a streaming engine; batch is realized as bounded streaming. This gives Flink superior performance and flexibility for both batch and streaming workloads.

Alibaba’s Blink and Future Directions

Alibaba contributed heavily to Flink, creating an internal version called Blink, which powers many of its real‑time services (search, advertising, security, etc.) and is offered as a cloud service. Future work includes deeper AI integration and merging Flink with micro‑service architectures.

Q&A Highlights

Why is Flink better suited for batch‑stream unification? Its streaming‑first design incurs little overhead when handling bounded streams.

Is Flink’s SQL more versatile than MPP engines? Flink SQL supports both short and long queries and provides robust failover.

Can DataSet and DataStream APIs be unified? A DAG API is being explored to express both batch and stream semantics.