StarRocks + Fluss: Query Real-Time & Historical Data in One SQL Statement

Real-Time Analytics Challenges Beyond Query Speed

As demand grows for real-time dashboards, business monitoring, and anomaly detection, analytics must reflect business changes promptly. Fast query execution alone is insufficient; newly generated data must appear in analysis results without delay. For example, during major promotions, operations teams need continuous observation of transaction changes across channels; when equipment anomalies occur, staff must correlate latest events with historical records. If data requires multi-layer synchronization and processing before querying, analysis results lag behind business reality even if queries themselves are fast.

StarRocks + Fluss + Paimon: A Unified Architecture

The combination of StarRocks and Apache Fluss provides a more direct path to reduce this wait. Fluss ingests continuously changing data, StarRocks supplies SQL query and analysis capabilities, and Apache Paimon handles long-term retention and historical backtracking. Together they allow querying real-time and historical data through a single logical table.

Architecture Overview

In this architecture, Flink handles data ingestion and stream processing, Fluss stores and provides real-time data access, Paimon manages long-term lake retention, and StarRocks delivers SQL query, join, and aggregation. An independent Tiering Service continuously tiers data from Fluss to the lake, so real-time consumption does not need to wait for lake table commits at each processing layer.

Figure 1: Real-time segment and lake data each play their role; StarRocks provides unified query entry. Lakehouse integration enabled when lake ingestion and permissions are configured.

Three Core Scenarios

Real-time business analysis: Incorporate newly arrived transaction, payment, or access events into statistics and observe alongside historical trends.

Real-time link troubleshooting: Use SQL to inspect log tables or processing results, verifying whether a batch of data has arrived or a field matches expectations.

Real-time and historical joint analysis: Device monitoring and business anomaly analysis need both new changes and long-range context.

Fluss vs. Kafka and Lake Storage

Kafka focuses on event delivery: pub/sub, service decoupling, consumption replay, and a rich client ecosystem make it a mature choice for messaging.

Fluss organizes stream data as tables. It provides log tables and primary key tables with columnar reads; primary key tables support updates, deletes, and primary-key-based access. These capabilities let continuously changing business data be accessed as tables by downstream processing tasks and analytical engines. For example, querying a wide table but needing only a few columns (channel, time, amount) allows column pruning to reduce data transfer; for orders with evolving state, primary key tables record updates so downstream sees the latest state.

Figure 2: Choose capabilities per business need; Kafka, Fluss, and lakehouse can also work together.

When combined with lake storage, Fluss provides direct access to the latest data. Paimon and other open lake formats handle long-term retention and historical queries. The Tiering Service continuously writes data to the lake and maintains synchronization progress between stream and lake storage. This division connects real-time access and historical queries in one architecture, reducing the effort of maintaining separate data copies and sync pipelines for different query needs.

Querying Fluss Data Directly via StarRocks Fluss Catalog

StarRocks accesses Fluss databases and tables through a Fluss Catalog, enabling SQL queries and analysis without first importing data into StarRocks internal tables. Prerequisites include network connectivity between StarRocks and Fluss clusters, connection address, authentication, and read permissions for relevant databases, tables, and underlying storage. For lakehouse queries, the target table must have lake ingestion enabled and readable lake snapshots available.

The following example creates a Fluss Catalog using SASL username/password authentication (replace placeholders with actual configuration):

CREATE EXTERNAL CATALOG fluss_catalog
PROPERTIES (
  "type" = "fluss",
  "bootstrap.servers" = "<Fluss连接地址>:9123",
  "fluss.option.client.security.protocol" = "SASL",
  "fluss.option.client.security.sasl.mechanism" = "PLAIN",
  "fluss.option.client.security.sasl.username" = "<用户名>",
  "fluss.option.client.security.sasl.password" = "<密码>"
);

Authentication must match the Fluss cluster. Beyond the bootstrap address, StarRocks must reach the data service nodes returned by Fluss.

Assume an append-only payment event table retail.payment_events partitioned by dt, containing channel and amount fields. The SQL below counts payments and sums amounts per channel for a specific date:

SELECT channel,
       COUNT(*) AS payment_count,
       SUM(amount) AS payment_amount
FROM fluss_catalog.retail.payment_events
WHERE dt = '2026-10-10'
GROUP BY channel
ORDER BY payment_amount DESC;

For tables with Paimon lake ingestion enabled, the default query merges lake data with real-time increments; for non-lake-enabled log tables, it reads directly from Fluss-retained data. After catalog creation, users can filter, aggregate, sort, and join with StarRocks internal tables or other sources. Results can be persisted via INSERT INTO ... SELECT ... into StarRocks internal tables for subsequent queries.

Union Read: Merging Real-Time Increments with Historical Lake Snapshots

This capability is implemented through Union Read (cold-hot data merge read) . At query time, the system reads lake snapshots and Fluss real-time increments not yet covered by those snapshots, based on recorded synchronization progress, and stitches the two parts together.

For log tables, merge read aligns the read boundaries of the two parts.

For primary key tables, it also merges updates and deletes for the same record to ensure primary key table update semantics.

Example: a lake snapshot records an order status as "pending payment", while the Fluss real-time increment has updated it to "paid"; the merge read result reflects the updated "paid" status.

Figure 3: User queries default table name; read path coordinates two layers; primary key tables also handle updates and deletes.

Merge read lets the latest non-lake data participate in queries, reducing analysis latency caused by waiting for lake ingestion. Data freshness and SQL execution time are measured separately: freshness depends on ingestion, processing, and query-visible progress; execution time depends on scan volume, join complexity, and resource configuration.

Inspecting Lake and Real-Time Parts Separately

Daily business analysis should use the default table name (no suffix) so the system coordinates real-time and historical reads. For lake-enabled append-only log tables, to inspect the lake snapshot and real-time increment separately, use the following query entry points:

-- View lake snapshot portion in this read plan
SELECT * FROM fluss_catalog.retail.`payment_events$lake` LIMIT 20;
-- View real-time increment after lake snapshot, within this query range
SELECT * FROM fluss_catalog.retail.`payment_events$rt` LIMIT 20;

Recent Optimizations

Support querying non-lake-enabled log tables: For real-time troubleshooting and short-cycle event analysis, directly query Fluss log tables without DataLake enabled, no need to build full lakehouse pipeline first. Lake ingestion can be configured later if historical range needs expansion.

Column pruning, partition pruning, and filter pushdown: Queries read only needed columns, narrow scan range via partition conditions, and push filters down to the lake side where possible, reducing data transfer and computation. Example: analyzing a specific date and channel combines partition and field filters to avoid reading irrelevant data.

Optimized Paimon historical data read path: For Paimon files meeting direct scan conditions, StarRocks uses a native read path to reduce cross-runtime data conversion overhead; for data requiring merging, it uses a merge read path to guarantee query results conform to table update semantics.

Connection and partition info reuse, count queries, empty scans, complex lake sharding: Continuous improvements reduce repeated initialization and metadata preparation overhead, enhancing stability across query scenarios.

Future Roadmap

Optimize real-time data reading: Explore native client and data exchange path optimizations to reduce overhead in real-time reads. Current Paimon native file reads mainly target lake-side data; native access for real-time side remains a future exploration.

Deepen compute pushdown and index utilization: Fluss ecosystem plans for filter/aggregation pushdown and index capabilities offer room to further reduce data scan and transfer. StarRocks will explore corresponding query optimizations as these capabilities evolve, with specific support to be implemented incrementally.

Extend open lake format support: Current lakehouse queries are based on Paimon. As more lake formats and unified access capabilities develop, future work may explore ingesting additional lake data types, expanding the unified query entry coverage.

Conclusion

The StarRocks and Fluss combination brings continuously changing data into SQL analysis and, through collaboration with Paimon, bridges real-time queries with historical backtracking. For teams already using StarRocks, this means they can continue using familiar analysis methods while getting the latest business changes into dashboards, anomaly detection, and join analysis more promptly. The related code has been merged into the StarRocks Main branch and will ship in the next release; subsequent articles will cover usage patterns and applicable scenarios as versions progress.