Fluss: Real-Time AI Data Storage Foundation — Primary Key Tables, Delta Join & Latest State Delivery

1. Problem Background: What Kind of Data Storage Layer Do Real-Time Intelligent Systems Need?

When a transaction enters a risk control system, the system may need to read the user's recent transaction frequency, cumulative amount, and current risk score to decide whether to approve. If the risk score comes from a slower-updating pipeline while transaction frequency comes from a separate real-time computation pipeline, the combined result may not represent a consistent business state even if each field is individually fast to query.

Real-time risk control, recommendation systems, and AI agents face three common problems: online requests need current state; training and serving need consistent data and feature definitions; historical analysis needs sufficient data to reconstruct past states.

Traditional architectures typically combine messaging systems, stream processing engines, online KV stores, and data lakes. These independent components are not inherently redundant: they may separately handle high-throughput ingestion, low-latency serving, and long-term history management. The trade-off is whether the cost of data replication, synchronization, and schema maintenance is worth the capabilities these components provide.

Apache Fluss offers a different data organization: using tables as the access abstraction, combining ordered logs with primary key state storage, and integrating with compute engines like Flink and open lakehouse table formats.

Positioning must be clear: Fluss is a distributed streaming storage system for real-time analytics and AI workloads. In the larger real-time intelligent data architecture, it belongs to the real-time data storage layer, not a complete data platform, feature platform, or AI serving platform. It can provide current state and change data to upper layers, but feature computation, model inference, business decisions, and full audit remain the responsibility of corresponding components.

The official architecture documentation describes Fluss as a streaming storage system that combines ordered logs with KV storage and supports remote storage and lakehouse integration.

This article focuses on three questions: How does Fluss maintain and serve the latest state? How does Delta Join affect streaming computation state costs? How can real-time storage, historical data, and training/serving be organized into a reliable data pipeline?

2. Architectural Positioning: Fluss Is the Real-Time Data Storage Layer, Not the Entire Data Infrastructure

In real-time intelligent systems, each component should have clear responsibility boundaries.

This is a logical division of responsibilities and does not mean an enterprise must deploy all components, nor that components are naturally compatible.

Fluss Internal Storage Structure

Figure 1: Fluss real-time data storage layer and its relationship with Lakehouse.

The local and remote storage in the diagram belong to Fluss's own data storage and recovery mechanisms; Lakehouse is another data representation for open table formats and historical analysis.

The two must not be confused:

Fluss remote storage can store tiered log segments and primary key state snapshots for historical log access and state recovery.

Lakehouse open table storage writes data into configured open table formats via the corresponding Tiering Service to support lake table queries and batch analytics.

Uploading Fluss log segments to object storage does not automatically convert them into Paimon or Iceberg tables; enabling Lakehouse integration triggers Flink Tiering jobs to write data into lake tables per configuration and maintain tiering progress.

Therefore, Fluss's core responsibility is real-time data storage and access; remote log tiering and Lakehouse integration are related but distinct capabilities.

3. Underlying Principles: Primary Key Tables, Delta Join, and Data Lifecycle

3.1 Primary Key Tables: Current State and Change Logs

Fluss supports Log Tables and Primary Key Tables.

Log tables suit continuous event appends, with records independently consumable within the retention window; primary key tables maintain current row state per primary key and record corresponding changes. For example, a user risk score may update from 0.12 to 0.47 to 0.83.

Primary key Lookup returns current state; change logs provide state change records while data remains within the retention window. Fluss provides virtual tables like $changelog and $binlog for accessing change information. Whether complete before/after images are available depends on virtual table type and configuration; not all change records carry full before/after values.

These two access modes solve different problems:

Must emphasize: Changelog does not equal complete decision audit. To explain why a transaction was rejected, beyond state changes you need to save decision timestamp, request ID, actual features used at that time, model and rule versions, decision result, and reason codes.

3.2 Delta Join: Reducing Redundant Data in Flink State

Traditional streaming joins often require maintaining input records in Flink State to match later-arriving data. As data scale grows, state storage, checkpoint, and recovery costs may rise.

Delta Join, when conditions are met, uses Fluss source table's indexed lookups to fetch matching records, reducing the need to redundantly store source table data in Flink State.

Figure 2: Delta Join converts part of join reads to source table lookups

Delta Join is not a stateless join; it still incurs join computation, lookup I/O, and downstream write costs. Whether benefits are realized depends on key design, data distribution, update frequency, and actual query plan.

Fluss official documentation states Delta Join is available from Flink 2.1; from Fluss 0.8, eligible streaming joins can be automatically optimized to Delta Join in corresponding versions.

Three aspects must be checked during design and troubleshooting:

Whether index and Join Key match. bucket.key must conform to table key constraints and match join conditions; arbitrary fields cannot be assumed to support generic secondary indexes.

Whether input and output semantics are supported. Join type, CDC change modes, and downstream Sink capabilities all affect optimization scope; different Flink versions have differences.

Whether the query plan actually uses Delta Join. Use EXPLAIN to check the actual execution plan, not just SQL syntax.

Delta Join's role is to reduce redundant computation state in applicable scenarios, not to replace all Flink State, nor to serve as a universal substitute for all streaming joins.

3.3 TTL, Remote Tiering, and Column Statistics: Managing Different Storage Costs

Data at different levels in Fluss has different lifecycles and should be configured separately.

In Fluss 1.0, table.kv.ttl can only be configured at primary key table creation, not dynamically modified via ALTER TABLE ... SET. Its cleanup is asynchronous and does not automatically emit DELETE events downstream.

Remote log tiering mainly addresses capacity, access, and recovery between local and remote storage; it is a different mechanism from writing to open lake tables. Data already tiered to the lake still needs lifecycle management by the lake table's own policies.

For access optimization, Fluss supports column statistics for Arrow log tables. Enabling table.statistics.columns allows the server to use batch-level min, max, and null statistics to skip data batches that cannot satisfy conditions, reducing unnecessary reads and transfers. This feature is limited by log format, configuration, and version, and does not automatically backfill statistics for historical data written before enabling.

Server-Side Scan targets specific interactive small-scale scans; unlike primary key Lookup, it is not suitable for replacing large-table bulk reads or high-concurrency scans.

The key to storage design is managing current state, change history, local hot data, and lake-side history separately, not using a single TTL to solve all data lifecycle problems.

3.4 Same Data Foundation Does Not Guarantee Training-Serving Consistency

Building real-time state and historical reads on a common data foundation helps reduce discrepancies caused by offline and online pipelines processing data separately. However, training-serving skew can also stem from event time vs processing time, feature logic, null handling, late events, and model version changes.

For example, training uses event-time windows while online serving uses processing-time windows; or training samples include data that arrives after the prediction timestamp. These issues occur even with the same underlying data.

Production systems still need unified feature definitions, explicit prediction timestamps, handling of late and duplicate events, and reconciliation of offline and online features.

Therefore, Fluss can provide a common real-time data storage foundation, but feature computation, historical sample reconstruction, and training-serving consistency must be jointly ensured by compute and model data pipelines.

4. Business Scenarios and Technology Selection: Which Problems Suit Fluss?

Take real-time risk control: Fluss can store account current risk state, Flink continuously updates features; but explaining a rejection decision requires independently saving the features, model version, request ID, and result at that moment.

Recommendation and ad systems can use primary key tables for current profiles, log or lake tables for behavior history. If downstream needs only a few columns of a wide table, evaluate column projection and column statistics pushdown; if online serving has stricter latency requirements, decide via load testing whether to retain a separate Serving layer.

Training data preparation cannot directly substitute current KV state for historical samples. Training tasks must correspond to correct prediction timestamps and avoid future information leakage; when historical range exceeds log retention window, samples must be constructed from lake-side historical data.

Thus, Fluss is more suitable for workloads requiring real-time state, continuous changes, and historical collaboration. Whether it can replace existing Redis, Feature Store, or other storage solutions should be decided by actual performance, functionality, and total cost.

5. Production Troubleshooting and Architecture Review: Locating Problems from Symptoms

Training performs well, online performance degrades.

Check offline vs online feature time semantics, default values, computation logic, and model versions, rather than attributing directly to storage performance. Especially verify training samples do not use information visible only after prediction time.

Primary key table state keeps growing.

Check table.kv.ttl, table.log.ttl, table.log.local-ttl, and partition retention policies separately. Do not treat row-level TTL as a precise deletion timer, nor assume TTL cleanup automatically generates downstream delete events.

Delta Join not effective or state remains large.

Verify Flink version, Bucket Key, join conditions, input change types, and downstream Sink; use EXPLAIN to check actual execution plan. Total state bloat may not come from Join but from other operators or caches.

Lookup latency increases.

Distinguish primary key Lookup, prefix lookup, and Server-Side Scan; analyze request distribution, server load, and network latency. Do not apply small-table scan strategies to large-scale reads.

Unable to reconstruct a specific decision.

Check whether Changelog still covers the target time range, and whether the audit system saved inputs, model and rule versions, results, and reason codes at that time. If logs have expired and history not archived, current state alone usually cannot recover the full decision process.

Architecture review ultimately must verify three things: whether the real-time storage layer meets current state access requirements, whether compute optimizations are validated by execution plans and load tests, and whether historical data and decision records satisfy rollback and audit requirements.

These three correspond to storage, compute, and data governance respectively; one component's capability cannot substitute for whole-architecture validation.

6. Summary: Fluss's Place in Real-Time Intelligent Data Architecture

The article uses a consistent responsibility boundary:

Fluss is the real-time data storage layer , responsible for logs, primary key state, and corresponding access capabilities.

Flink is the compute layer , responsible for continuous processing, feature updates, and streaming joins.

Open lake tables are the long-term historical data layer , carrying data suitable for historical queries, batch analytics, and training.

Online serving handles inference and business response , choosing appropriate data access methods per actual SLA.

Governance and audit systems enforce and record access and decisions , not replaceable by Fluss's storage capabilities alone.

Primary key tables provide current state and change log views; Delta Join reduces redundant data in Flink State under applicable conditions; TTL, remote tiering, and column statistics manage lifecycle and access costs respectively. These mechanisms make Fluss an important data foundation component for real-time intelligent systems, but do not automatically solve training-serving skew, full historical reconstruction, or complete decision audit.

Providing the latest state to AI is not just about letting it quickly fetch a value, but also clarifying how that value is produced, when it is valid, and whether it can be correctly reconstructed from retained data in the future.

Fluss handles the real-time data storage and access foundation; the rest — compute, serving, historical management, and governance — must be collaboratively designed in the overall architecture.

Next article (Part 27): "Paimon AI-Native Lakehouse — From Unified Lake Format for Data + AI to Multimodal Ingestion," continuing discussion on how open lake tables undertake long-term historical data management and how real-time storage layer and lake storage layer collaborate under clear responsibility boundaries.