Apache Fluss Graduates to TLP: Lakestream Unifies Streaming & Lakehouse

Background: Apache Fluss Graduates to Top-Level Project

On August 6, 2026, the Apache Software Foundation (ASF) announced that Apache Fluss has graduated from the incubator to become a Top-Level Project (TLP). The graduation proposal passed the Incubator PMC unanimously and was approved by the ASF Board. Fluss, initiated by the Alibaba Cloud Flink team, implements the "Lakestream" (湖流一体) architecture, now established as a mainstream standard for streaming lakehouses globally.

From Internal Project to Apache TLP: A Three-Year Journey

July 2023: Alibaba Cloud Flink team starts internal project to address analytical streaming storage, unified stream/batch storage, and Flink state management pain points.

November 2024: Open-sourced at Flink Forward Asia after large-scale production validation in Alibaba core e-commerce scenarios.

June 2025: Enters Apache Incubator for community governance.

July 2026: Incubator unanimously approves graduation; ASF Board ratifies.

August 2026: Official public announcement of graduation.

The name "Fluss" comes from German for "river," reflecting the vision of Flink Unified Streaming Storage: data flows continuously like a river into an open lakehouse.

Why Traditional Architectures Cannot Unify Real-Time and Historical Data

1. Lambda Architecture: Redundancy by Design

Kafka stores real-time hot data + lakehouse stores historical cold data. This appears to balance real-time and batch but introduces double trouble:

Data stored twice → storage cost doubles.

Two separate codebases for stream and batch → business metric drift causes inconsistency.

Operations must maintain message queue, compute engine, and lakehouse → complexity grows exponentially.

2. Pure Lakehouse: Inherent Real-Time Deficiency

Lakehouse formats (Paimon, Iceberg) rely on checkpoint snapshots; data freshness is naturally minutes-level. Multi-stage processing pipelines can push latency to 10+ minutes, insufficient for sub-second real-time risk control or recommendation scenarios.

3. Kafka Was Never Designed for Analytics

Row-oriented message storage lacks column pruning and partition pruning → analytical query efficiency is extremely low. No native primary key updates or point queries → Flink dimension joins must load entire dimension tables into TaskManager memory, leading to "state bloat → checkpoint timeout → task avalanche" death spiral.

Teams keep stacking components to compensate for storage-layer capability gaps, making architectures increasingly bloated.

Fluss's Answer: One Table, One View, Seamless Hot/Cold Integration

The Lakestream architecture uses a single tiered storage to carry all data from sub-second real-time to full history, presenting only one logical table to upper layers.

Hot/Cold Tiering: Separation of Concerns, Automatic Flow

The architecture splits into two layers; data flows from hot to cold automatically, completely transparent to business.

Hot Layer: Fluss Real-Time Storage Cluster

Comprises Coordinator (metadata management, shard scheduling) and multiple TabletServers (data read/write). Data stored locally in Apache Arrow columnar format for high-frequency hot data.

Log Table: Append-only, analogous to Kafka Topic, optimized for high-throughput streaming write and consume.

Primary Key Table: Native Upsert updates and row-level deletion, built-in RocksDB KV engine, sub-second point queries, directly replaces Redis for dimension storage.

Hot layer retains hours to days of hot data, millisecond read/write latency, serving all real-time access traffic.

Cold Layer: Open Lakehouse Formats

Built-in Tiering Service asynchronously compresses and converts expired hot data to Parquet, automatically archiving to open lakehouse formats (Paimon, Iceberg, Hudi, Lance) for full historical data, supporting offline analysis and AI training.

True Unification: Unified Metadata + Union Read

Unified Metadata: Hot and cold layers share the same table schema, partitioning, and bucketing rules. DDL changes sync automatically; business maintains only one logical table.

Union Read: Query engines access the same table; Fluss automatically merges hot-layer real-time increments with cold-layer historical archives. Deduplication, alignment, and stitching happen entirely in the storage layer, fully transparent to upper compute.

Query real-time data → hot layer; query historical data → cold layer; query full data → automatic merge. No table switching, no UNION ALL, one dataset, one metric, seamless coverage from seconds to full history.

Four Core Capabilities Rebuilding the Real-Time Data Foundation

1. Columnar Streaming Storage: Better Than Kafka for Analytics

Based on Apache Arrow columnar storage, server-side natively supports column pruning, predicate pushdown, and partition pruning. Streaming consumers fetch only required fields, reducing network I/O and memory overhead by multiples compared to row-oriented Kafka, natively fitting analytical streaming workloads.

2. Native Primary Key Updates + KV Point Query: Freeing Flink Large State

Primary Key Tables support row-level Upsert and partial column updates, with built-in KV engine for sub-second point queries. Flink dimension Lookup Joins no longer need to load full dimensions into task state; they query Fluss Primary Key Tables directly.

TaskManager memory and CPU usage drop significantly.

Checkpoints no longer wait for large state flushes; alignment time plummets.

Task failure recovery completes in seconds, no need to replay tens of GBs of state.

This is why Fluss solves checkpoint timeouts at the root: moving state from compute nodes back to storage, making compute stateless again.

3. Unified Stream/Batch Reading: One API for All Scenarios

The same table supports both streaming incremental consumption (like Kafka) and batch full scans (like lakehouse tables). Streaming jobs and batch jobs reuse the same table and data; metrics are naturally consistent, eliminating dual pipeline maintenance.

4. Fully Open Ecosystem: No Vendor Lock-In

Fluss pursues openness from day one:

Compute Engines: Deep integration with Flink, Spark; compatible with StarRocks, Trino, Doris, DuckDB, Ray.

Lakehouse Formats: Supports Paimon, Iceberg, Hudi, Lance; seamless integration with existing lakehouse foundations.

Multi-language SDKs: Java, Python, Rust/C++ covering diverse business stacks.

Fluss does not aim to build another closed ecosystem but to provide the missing real-time hot layer puzzle piece for existing lakehouse architectures.

Fluss Does Not Replace Kafka or Paimon — It Complements Them

For most teams with existing architectures, the optimal path is not rip-and-replace but adding a Fluss hot layer on top of the current lakehouse .

Taking the common "Flink + Paimon + StarRocks" stack as an example, migration is smooth:

Flink changes from dual-write to Paimon + StarRocks → single write to Fluss.

Real-time queries: StarRocks directly queries Fluss hot layer, maintaining sub-second latency.

Historical archival: Tiering Service automatically sinks data to Paimon.

Full queries: StarRocks uses Union Read to transparently access hot+cold full data.

No dual-write redundancy, no consistency risk, Flink state pressure drops sharply, and the entire architecture transforms from bloated to clean.

Four Scenarios First to Benefit from Streaming Lakehouse

1. Real-Time Data Warehouse & Analytics

Second-latency operational dashboards and real-time monitoring boards replace Lambda architecture with a single Fluss + lakehouse stack, cutting pipeline length in half and guaranteeing metric consistency.

2. Real-Time Risk Control & Recommendation

Sub-second writes + KV fast queries support real-time rule validation and online feature serving, eliminating the need for Redis + offline data warehouse patchwork.

3. AI Feature Engineering & Agent Context

Real-time features live in Fluss for online serving; historical features sink to lakehouse for offline training. AI Agents retrieve complete data context from history to real-time via a single table for more precise decisions.

4. IoT Time-Series & Device Monitoring

Device second-level signals trigger real-time alerts; long-term data auto-archives for fault modeling and analysis. One storage layer handles both real-time alerting and offline training.

Conclusion

From data warehouse to data lake to lakehouse, data infrastructure evolution always moves toward "simpler, more unified, more real-time." Lakehouse unified "batch and history"; Lakestream unifies "real-time and history." Fluss's graduation is not the finish line but the starting point for streaming lakehouse adoption. When real-time and history are no longer two disconnected pipelines, when one dataset serves both sub-second response and full-scale analysis, when we no longer stack endless components for consistency and performance — we take a giant step toward the ideal of "data available anytime." After all, the best architecture is always the simpler one.

Reference: [1] Fluss Community: Apache Fluss Graduates to a Top Level Project