Flink 2025 Updates: Disaggregated State, AI Agents, and SQL Enhancements

Disaggregated State Management & Cloud‑Native Architecture

Flink 2.0 introduces a Disaggregated State Management architecture that separates state storage from the compute layer. State is persisted in remote durable storage such as HDFS or Amazon S3, while TaskManagers keep only a small cache. This eliminates container‑level disk limits, removes CPU/IO spikes caused by compaction, and reduces the time and risk of state migration during scaling.

Asynchronous Execution Model

The new model decouples state access from the processing thread. When a record triggers a state read or write, the operation is issued asynchronously; the processing thread proceeds to the next record without blocking. This non‑blocking behavior improves throughput and latency, especially when state resides in remote storage.

ForSt – Hierarchical Storage Backend

ForSt is a layered storage system designed for the disaggregated architecture. Each TaskManager maintains a two‑level cache (in‑memory and local‑disk) that holds hot state, providing near‑local latency. All state that does not fit in the cache is flushed to the remote store, enabling checkpoint durations of under 10 seconds even for terabyte‑scale state.

Performance and Stability Optimizations

Adaptive Batch Execution – the batch runtime automatically adjusts execution plans and resource allocation based on observed data characteristics, reducing unnecessary resource consumption.

Universal Incremental Checkpointing – only the changed portions of state are transferred during a checkpoint, dramatically shrinking network traffic and checkpoint latency.

High‑speed Network Tuning – serialization and transport protocols are optimized for 25 Gbps+ networks, achieving performance comparable to local SSDs.

AI Integration

Flink Agents

Flink Agents is a new sub‑project that provides an agent‑programming framework for event‑driven AI agents. It bundles abstractions for large language models (LLMs), memory, tools, and prompts, and adds capabilities such as dynamic execution plans, loop execution, shared state, and observability. Native connectors are provided for major LLM providers (e.g., OpenAI) and vector databases (e.g., Milvus). Users can also plug in custom internal models.

Deep SQL‑AI Integration

Flink 2.0 adds a Model DDL syntax. The CREATE MODEL statement lets users register, version, and manage AI models directly in the Flink catalog, enabling model inference to be invoked inside streaming SQL queries.

SQL Engine Enhancements

Materialized views for incremental result caching.

Delta Join – replaces full‑state stream joins with an external index (key‑value store) plus changelog streams, moving most join state out of Flink and avoiding state explosion.

Multi‑Join support for joining more than two streams in a single query.

New VARIANT data type for efficient handling of semi‑structured (JSON‑like) data.

Process Table Functions (PTFs) – allow stateful read/write operations inside SQL, enabling complex, stateful computations.

Additional Improvements

Extended monitoring metrics and tighter integration with PyFlink for better observability and Python API performance.