Apache Fluss: Dual Engines & Tiered Storage Power Real-Time Lakehouse

Overall Architecture: Classic Master-Worker Layered Architecture

Apache Fluss adopts the mature and efficient Master-Worker architecture pattern, divided into four layers: control plane, data service layer, remote storage layer, and client layer. This design embodies storage-compute separation, horizontal scalability, and high availability fault tolerance, with clear responsibilities suitable for large-scale production clusters.

Each layer has distinct responsibilities: the client layer handles business interactions, the control plane manages cluster scheduling and governance, the data layer serves actual read/write operations, and the remote storage layer handles low-cost cold data archiving, together forming a complete stream-lake integration loop.

Core Components Deep Dive

1. CoordinatorServer: Cluster Global "Brain"

CoordinatorServer is the control plane core, responsible for cluster metadata management, resource scheduling, high availability assurance, and lifecycle control. As a stateless component, it can be deployed in a cluster with multi-node redundancy to ensure uninterrupted scheduling.

2. TabletServer: Core Data Read/Write Service Node

TabletServer is the data plane core; all data writes, reads, updates, and archiving are handled by this node cluster. Unlike traditional storage architectures, each TabletServer embeds both LogStore and KvStore dual storage engines, which are activated on demand based on table type for precise functional and performance adaptation.

2.1 LogStore: Streaming Log Storage Engine

LogStore draws on database binlog and Kafka log design principles, providing low-latency streaming capabilities with high throughput, low latency, and high reliability:

Pure append-only write mechanism : Supports only Append writes; data is immutable once written, ensuring streaming data integrity and ordering.

Sub-second streaming read optimization : Optimized for real-time incremental consumption scenarios, suitable for Flink real-time computing and dashboarding.

WAL pre-write log capability : Serves as the fault recovery foundation for KvStore; all updates are first written to LogStore, enabling data reconstruction via log replay during failures, guaranteeing no data loss.

2.2 KvStore: Key-Value Update Storage Engine

KvStore matches database transaction capabilities, filling the update gap in streaming storage and serving as the core for real-time wide tables and dimension table joins, with full CRUD support:

Real-time updates and deletes : Native support for primary key Upsert, incremental updates, and precise deletes, ideal for CDC change data and business real-time updates.

High-concurrency point queries : Supports high QPS low-latency key-value lookups, can directly replace Redis as Flink Lookup Join dimension tables, simplifying architecture.

Complete Changelog generation : All data changes automatically generate incremental logs, seamlessly consumed by downstream computing and archiving services for end-to-end data pipeline connectivity.

2.3 On-Demand Engine Activation: Scenario-Adaptive Performance Optimization

One of Fluss's most elegant designs is automatically activating the corresponding storage engine based on table type, avoiding unnecessary resource overhead and maximizing scenario adaptation.

3. Tablet/Bucket: Fine-Grained Data Sharding Mechanism

Fluss organizes data through a clear hierarchical model using bucketing and sharding to achieve uniform data distribution and horizontal scaling. The data hierarchy is as follows:

Core design principles and high availability features:

Data locality guarantee: LogTablet and KvTablet for the same Bucket are co-located on the same TabletServer, avoiding cross-node read/write overhead and improving performance.

Streaming data high availability: LogTablet supports multi-replica deployment, ensuring streaming data durability and service continuity through replica redundancy.

Architecture evolution plan: Current version KvTablet does not yet support replicas; it relies on remote storage snapshots plus log replay for fault tolerance. Future versions will add Kv replicas to further improve availability.

4. ZooKeeper: Cluster Coordination Foundation

In the current stable version, Fluss relies on ZooKeeper for core cluster coordination, providing three fundamental capabilities: cluster node heartbeat coordination, global metadata persistent storage, and unified cluster configuration management.

Architecture Evolution Plan : Future versions will completely remove the ZooKeeper hard dependency, replacing it with a self-developed KvStore for metadata and the Raft protocol for distributed coordination, significantly reducing operational complexity and improving cluster stability and scalability.

5. Remote Storage: Core Support for Lake-Stream Unification

Remote storage (OSS/S3/HDFS) is key to Fluss's tiered hot/cold storage, storage-compute separation, and low-cost archiving, serving both LogStore streaming tiering and KvStore persistence scenarios, bridging the gap between real-time streams and offline lakes.

5.1 LogStore Hot/Cold Tiered Storage

Hot data resides on local high-speed SSDs, guaranteeing sub-second real-time read/write and incremental consumption performance.

Expired cold data automatically sinks to low-cost object storage without manual intervention, drastically reducing cluster storage costs.

Optimizes cluster scaling efficiency: historical cold data requires no migration; only metadata synchronization is needed for node expansion, avoiding massive data relocation overhead.

5.2 KvStore Persistence and Failure Recovery

KvStore incremental data periodically generates snapshots persisted to remote storage for long-term retention.

Builds a dual fault-tolerance mechanism of "remote snapshots + LogStore log replay", enabling rapid full data recovery after node failures to ensure business continuity.

5.3 Efficient Batch Read/Write Capability

Supports clients directly batch-reading historical data from remote storage, bypassing TabletServer compute pressure and greatly improving offline analysis and data backtracking efficiency; future iterations will add batch write capabilities for large-scale data initialization import scenarios.

6. Client Layer

Fluss currently centers on the Flink Connector as the primary client interaction entry point, deeply integrating with the real-time computing ecosystem and supporting full-scenario data operations: streaming read/write, batch read/write, DDL schema operations, and high-concurrency point queries. It fully aligns with Flink SQL development paradigms, allowing zero-cost onboarding. The community is also rapidly iterating Spark, StarRocks, and other multi-engine clients to continuously enhance the open ecosystem.

Core Architectural Design Highlights Summary

Fluss's architecture balances performance, cost, stability, and scalability, each design precisely addressing traditional big data architecture pain points:

Core Competitiveness Summary

Overall, Apache Fluss builds unique competitiveness distinct from traditional streaming storage, data lakes, and OLAP engines through its differentiated underlying design:

1. Truly Unified Real-Time Data Foundation : LogStore solves low-latency streaming data transport; KvStore solves real-time updates and state point queries. The dual-engine collaboration covers all real-time data warehouse scenarios, replacing Kafka+Redis combinations and simplifying architecture.

2. Native Lake-Stream Unification Closed Loop : Leveraging remote tiered storage, hot data serves real-time needs while cold data automatically archives into the lake, seamlessly connecting real-time and historical data without additional sync jobs.

3. Multi-Layer High Availability Guarantees : Log multi-replica redundancy, remote storage snapshot persistence, and WAL log replay provide three layers of protection, ensuring data reliability and service stability.

4. Full Ecosystem Open Compatibility : Deeply bound to the Flink ecosystem, compatible with mainstream computing, query, and lakehouse engines, enabling seamless integration into existing enterprise big data architectures without intrusion.

Conclusion

Apache Fluss's Master-Worker layered architecture, dual storage engine adaptive design, and tiered lake-stream unification mechanism reconstruct the technical foundation of real-time lakehouses from the bottom up. It compensates for traditional streaming storage's inability to efficiently update and point-query, while solving data lake real-time performance gaps. Through storage-compute separation, on-demand adaptation, and automatic archiving, it perfectly balances four core demands: real-time performance, storage cost, architectural simplicity, and data reliability.

As real-time data warehousing, lakehouse integration, and AI real-time feature scenarios rapidly proliferate, Fluss's lightweight, highly elastic, and highly unified underlying architecture has become one of the optimal choices for next-generation enterprise real-time data infrastructure.