Building a Lake‑Stream Integrated Data Pipeline with Fluss, Paimon, and StarRocks

In response to growing real‑time business demands and shrinking decision‑making cycles, Taotian Group faced two core challenges: (1) rapid generation of real‑time data that could not be easily incorporated into a unified, governed, and queryable data system, and (2) lakehouse data that offered strong governance and analytical capabilities but suffered from minute‑level or higher latency.

The goal was to enable second‑level real‑time data, minute‑level lakehouse data, and offline historical data to flow continuously within a single system.

Component Roles

Fluss : Provides second‑level real‑time storage and consumption, offering log tables, primary‑key tables, column pruning, and multi‑level partition pruning.

Paimon : Serves as the lakehouse storage layer, handling minute‑level data ingestion, historical data management, and stream‑batch unified computation results.

StarRocks : Acts as the unified query service, merging Fluss’s second‑level increments with Paimon’s minute‑level and historical data into a single OLAP path, reusing Paimon’s table‑scan, predicate‑pushdown, and complex query optimizations.

In the new architecture, all three data layers (ODS, DWD, ADS/DWS) can be backed by Fluss for second‑level real‑time data, while Fluss automatically synchronizes data to Paimon to form minute‑level lakehouse tables. Offline batch data continues to write to Paimon’s historical branch.

Fluss Real‑Time Ingestion

Fluss supports two table models:

Log Table : Append‑only, similar to Kafka/TimeTunnel, no updates or deletes, achieves peak write throughput of 35 million rows per second.

Primary‑Key Table : Supports INSERT, UPDATE, DELETE for stateful data such as transactions or user status, with peak write throughput of 5 million rows per second. The LastRow engine enables partial column updates, allowing multiple streams to merge into a wide real‑time table without overwriting unchanged fields.

Fluss also reduces consumption cost through:

Column Pruning : Downstream tasks read only required fields (e.g., id, status), cutting bandwidth by over 90% for wide columns.

Multi‑Level Partition Pruning : Tasks read only target partitions, avoiding full‑table scans.

Configuration for Lake‑Stream Integration

Enabling the integration is done via table parameters: table.datalake.enabled = true: Turns on lake‑stream mode, automatically creates a matching Paimon table. table.datalake.freshness: Controls how often Fluss writes to Paimon (default 3 minutes, adjustable per latency requirements). paimon.file.format: Sets the file format for Paimon tables (e.g., Parquet).

Query Path in StarRocks

During a query, StarRocks splits the data access based on Fluss checkpoints:

Before checkpoint : Reads from Paimon, leveraging its mature scan, column‑pruning, and predicate‑pushdown optimizations for most historical and minute‑level data.

After checkpoint : Reads the latest seconds of data directly from Fluss, keeping additional latency minimal.

Business view : The split is transparent to users, who see a single OLAP query path.

This design combines the high‑throughput of lakehouse scans with the low‑latency of real‑time streams, eliminating the need for manual data movement or stitching.

Results and Future Work

After phased deployment, the lake‑stream pipeline moved from validation to production, delivering concrete benefits:

Development and operations efficiency improved by >50%, reducing verification cycles from 5 days to 2 days.

Real‑time query latency reached second‑level response.

Consumption bandwidth and deserialization cost dropped by >80% thanks to column and partition pruning.

Planned next steps include:

Broadening Fluss adoption across the group to further lower real‑time consumption cost.

Building real‑time materialized views that auto‑incrementally refresh in StarRocks when Fluss data changes.

Exploring AI‑driven real‑time rule engines that ingest Fluss streams for anomaly detection and immediate business alerts.

Overall, the integration demonstrates that by schema‑fying second‑level data with Fluss, persisting minute‑level and historical data with Paimon, and providing a unified high‑performance query layer with StarRocks, a seamless lake‑stream architecture can be achieved, delivering both low latency and robust analytical capabilities.