How Flink Agents and Flink 2.0 Are Powering Real‑Time AI at Scale

From Real‑Time Data Analytics to Real‑Time AI, Flink Ecosystem Fully Embraces AI

At Flink Forward Asia 2025, Wang Feng, initiator of the Apache Flink Chinese community and PMC member of Apache Paimon, presented the talk “From Real‑Time Data Analytics to Real‑Time AI”. He emphasized that the rise of Agentic AI makes the combination of real‑time data and real‑time AI a critical component for large‑scale production AI applications.

System‑triggered AI agents —beyond user‑initiated agents—will increasingly be driven by events such as online transactions, website clicks, vehicle status changes, or IoT sensor updates, demanding higher compute scale, stability, and deep integration of real‑time data processing with AI.

To address this, the Apache Flink community launched a new sub‑project called Flink Agents . Flink Agents is an agent‑programming framework designed for event‑driven AI agents. Built on the Flink streaming engine, it inherits large‑scale, distributed, real‑time processing, state management, consistency guarantees, and fault‑tolerance. It also provides abstractions for LLM, memory, tools, prompts, dynamic execution plans, looping, shared state, and observability. The project is contributed by Alibaba Cloud, Confluent, Ververica, LinkedIn, and others, with an MVP expected around September.

Flink 2.0: Disaggregated State Management and Cloud‑Native Architecture

Flink 2.0 introduces a Disaggregated State Management architecture that separates state storage from compute tasks, leveraging cheap object storage for shared data. This enables more flexible resource scheduling, higher scalability, and lightweight, stable fault tolerance. It fundamentally solves long‑standing issues of large snapshot overhead, slow state recovery, and high cost caused by tightly coupled state and compute.

The research paper “Disaggregated State Management in Apache Flink® 2.0” was co‑authored by the Flink community, Alibaba Cloud’s real‑time computing team, and academic researchers, and has been accepted by VLDB 2025.

Paimon: Multi‑Modal Unified Lake Storage for the AI Era

Apache Paimon, a streaming‑batch unified storage system, integrates with Flink to build a streaming lakehouse. With Iceberg V3’s Deletion Vectors, Paimon can sync Iceberg snapshots in real time, achieving minute‑level query latency. It also adopts the Lance file format to efficiently store large binary objects (e.g., audio‑video), supporting multi‑modal data. Deployed internally at Alibaba, Paimon processes hundreds of petabytes and up to 40 million rows per second per table, and is used by Vivo, Xiaomi, ByteDance, Shopee, and others.

Alibaba Cloud has integrated fully managed Paimon into its Data Lake Fabric (DLF) product, reducing storage costs by over 30 % and doubling query performance. The new Paimon Catalog is now in public beta in Singapore and Jakarta.

Fluss: Real‑Time Data‑Analysis and AI‑Workload Table Storage

Fluss, an open‑source table storage system from Alibaba, combines columnar storage with streaming updates and integrates tightly with Flink and lakehouse formats such as Paimon and Iceberg. It lowers the cost of building real‑time data warehouses and improves developer productivity through unified batch and stream capabilities, partition pruning, and zero data duplication.

Since its open‑source release in December 2024, Fluss has attracted contributors from ByteDance, Ant Financial, Xiaomi, eBay, Tencent, Dream11, and others. In June 2025, Alibaba donated Fluss to the Apache Software Foundation, marking its transition to a neutral, community‑driven project.

Industry Perspective

Mike Gualtieri, Vice President of Forrester, noted that Apache Flink has become the de‑facto standard for real‑time data processing, serving as the central nervous system for AI‑enabled enterprises. Real‑time streams allow organizations to integrate diverse data sources, support event‑driven architectures, and build real‑time AI agents, confirming Flink’s roadmap “The Future of AI is Real‑Time”.