Apache Fluss Dual-Table Model: LogTable and PrimaryKeyTable Explained

Overall Data Organization Model

Fluss adopts a layered data organization from logical to physical:

Database → Table → Partition → Bucket → Tablet → Segment

Each layer handles different responsibilities: namespace isolation, horizontal scaling, and operational efficiency.

1.1 Hierarchy Overview

Database : Logical collection of tables providing namespace isolation, creation, deletion, and permission management.

Table : Core storage unit organized by rows and columns. Two types exist based on primary key definition: LogTable and PrimaryKeyTable.

Partition : Logical split by one or more partition columns. Enables data isolation, lifecycle management, and query pruning. Supports manual, automatic (hour/day/month/quarter/year), and dynamic strategies. Constraint: PrimaryKeyTable partition columns must be a subset of the primary key.

Bucket : Smallest parallel unit for horizontal data splitting, also the minimal physical unit for migration and backup. A table/partition splits into N buckets via routing strategies.

Tablet : Real storage carrier inside a bucket, two types:

LogTablet (all tables): Based on Apache Arrow columnar storage, handles WAL, streaming changes, and changelog storage. Comprises multiple Segments, each with .index offset index and .log data files.

KvTablet (PrimaryKeyTable only): Backed by an independent RocksDB instance (LSM-Tree), responsible for updates, deletes, and primary key point lookups.

Segment : Physical file fragment inside LogTablet, supports auto-rolling, compression, and cold/hot tiering.

1.2 Architectural Design Principles

Data Locality : LogTablet and KvTablet of the same bucket always schedule to the same TabletServer, avoiding cross-node read/write overhead.

Compute-Storage Separation : Hot data resides on local SSD for low-latency access; cold data automatically sinks to object storage. Scaling does not require full data migration.

Capability Reuse : Both table types share partition management, cold/hot tiering, metadata scheduling, and lakehouse sinking, eliminating architectural redundancy.

LogTable: Pure Append Streaming Log Table

LogTable targets high-throughput, immutable, pure streaming scenarios, benchmarking against traditional message queues but with generational upgrades in storage structure and analytical performance.

1. Core Positioning & Applicable Scenarios

LogTable only supports INSERT append writes; data cannot be modified or deleted after write, strictly preserving write order. Typical scenarios: user behavior tracking, system logs, audit trails, ODS raw incremental data, IoT sensor reports.

2. Standard DDL Syntax

Omitting PRIMARY KEY creates a LogTable. Bucket count specified via bucket.num:

CREATE TABLE log_table (
  order_id BIGINT,
  item_id BIGINT,
  amount INT,
  address STRING,
  dt DATE
) WITH ('bucket.num' = '3');

3. Three Bucket Routing Strategies

Sticky (default) : Randomly picks a bucket, writes in batches until full, then switches. Minimizes connection switching overhead, optimal throughput for non-strict ordering.

Round-Robin : Distributes each record sequentially across buckets, achieving perfectly even load distribution, ideal for skewed data distributions.

Hash Routing : Uses bucket.key to specify a business field; hashes the field value to route to a bucket. Guarantees write order per business key.

4. Streaming Consumption Ordering Semantics

Within a single bucket : Strictly follows write order; earlier writes consumed first. Semantics identical to Kafka partitions.

Across buckets : Multi-threaded parallel consumption; no global ordering guarantee.

5. Columnar Storage Core Advantages

Native Streaming Column Pruning : Consumers and queries read only required fields, avoiding full record parsing. Official benchmarks show up to 10x read performance improvement and significantly reduced network transfer.

Independent Column Compression : Supports ZSTD/LZ4_FRAME (default ZSTD Level 3), average compression ratio >5x. Each column compressed independently while retaining efficient column pruning.

Vectorization Friendly : Columnar memory layout naturally aligns with SIMD vectorized computation, providing performance foundation for downstream analytical engines.

PrimaryKeyTable: Updatable Primary Key State Table

PrimaryKeyTable targets business state updates, dimension joins, real-time aggregation , unifying streaming log and KV state storage to fill the update gap of traditional streaming storage.

1. Core Positioning & Applicable Scenarios

Guarantees data uniqueness via primary key, fully supports INSERT/UPDATE/DELETE. Stores latest business snapshot state, replacing the traditional "message queue + cache + state store" combo. Typical scenarios: MySQL/PostgreSQL CDC sync, real-time order wide tables, user profile dimension tables, streaming pre-aggregation, high-cardinality dictionary mapping.

2. DDL Rules & Core Constraints

Declare primary key via PRIMARY KEY ... NOT ENFORCED; NOT ENFORCED means logical constraint enforced by upstream:

CREATE TABLE pk_table (
  shop_id BIGINT,
  user_id BIGINT,
  num_orders INT,
  total_amount INT,
  PRIMARY KEY (shop_id, user_id) NOT ENFORCED
) WITH ('bucket.num' = '4');

Mandatory Production Constraints :

Multiple writes for same primary key retain only the latest version.

If partitioning enabled, partition columns must be a subset of the primary key to prevent cross-partition key conflicts.

Bucketing only supports primary key hash routing; random strategies disabled to ensure same key always lands in same bucket.

3. Fixed Hash Bucketing Mechanism

Unlike LogTable's flexible strategies, PrimaryKeyTable uses fixed primary key hash routing : default bucket key is the primary key minus partition columns. This ensures each bucket maps to an independent RocksDB instance, enabling efficient Update/Delete and precise point lookups.

4. Core Capabilities Detailed

(1) Partial Update

Multiple business pipelines can update different columns independently without upstream Flink multi-stream joins. Fluss automatically merges new columns with existing data to produce the latest complete record. Only requirement: write must carry full primary key. This dramatically simplifies real-time wide table development and reduces compute cluster state pressure.

(2) Pluggable Merge Engines

Users can define custom merge rules for same-key records. Four built-in engines cover all scenarios:

Default Last-Row : Keeps last update; fits most business state wide tables.

First-Row : Keeps first write; fits first-behavior/first-order statistics.

Versioned : Uses a specified version field; retains highest version; fits versioned business data.

Aggregation : Supports SUM, MAX, MIN, RoaringBitmap etc.; auto pre-aggregates on write; fits real-time UV, cumulative metrics.

(3) Complete CDC Changelog

All insert/update/delete operations automatically generate standard CDC changelog with four operation types: +I (insert), -U (update before), +U (update after), -D (delete). Changelog reuses LogTablet storage, natively supports streaming column pruning. Downstream can consume changes at low cost for sync, distribution, and computation, achieving "state queryable, changes traceable, history replayable".

(4) Auto-Increment Column & Streaming Dictionary Table

PrimaryKeyTable supports auto-increment fields generating globally unique integer IDs, solving high-cardinality string ID deduplication performance issues. Mapping strings to compact integers combined with native rbm32/rbm64 bitmap aggregation boosts massive DISTINCT performance by multiples to orders of magnitude.

Performance Design : Each bucket locally caches a batch of IDs (default 100,000) for allocation efficiency; thus auto-increment IDs do not guarantee strict global monotonic increase, only global uniqueness and rough chronological order .

Best Practice : Combine with Flink Lookup Join's lookup.insert-if-not-exists parameter to fully automate dictionary dimension table construction in streaming jobs without offline pre-processing.

(5) Multi-Dimensional Query Capabilities

Streaming Read : Default reads full snapshot then continues with incremental changelog; can also consume only incremental changes.

Exact Lookup Point Query : Sub-second KV lookup via full primary key; can directly replace Redis for Flink dimension joins.

Prefix Lookup : Supports composite primary key prefix batch scans; fits range queries and batch joins.

Dual-Table Comparison & Production Selection Guide

1. Core Capability Comparison

2. Production Selection Principles

Prefer LogTable when :

Data is append-only with no update/delete logic.

Scenarios: logs, tracking, audit trails, ODS raw increments, high-throughput pure streaming.

Core demand: extreme throughput, low storage cost, low consumption I/O.

Prefer PrimaryKeyTable when :

CDC sync from business databases with frequent updates/deletes.

Need real-time dimension joins, high-frequency primary key point lookups.

Multiple async streams update different columns; need storage-layer auto-merge.

Complex state scenarios: real-time pre-aggregation, bitmap deduplication, streaming dictionary mapping.

Unified Architectural Foundation: Shared Underlying Capabilities

Despite significant capability differences, LogTable and PrimaryKeyTable fully share the same storage architecture and core mechanisms, key to Fluss's simplicity and operability:

Unified Cold/Hot Tiering : Hot data on local SSD for low latency; cold data auto-sinks to object storage with TTL-based local cleanup.

Unified Lakehouse Sinking : Both table types can automatically archive to Paimon/Iceberg data lakes via Tiering Service without extra sync jobs, achieving stream-lake integration.

Unified Query View : Union Read automatically merges local hot data and remote cold data; business gets a complete unified view without awareness of data location.

Unified Metadata Scheduling : Shared CoordinatorServer for metadata management, resource scheduling, and failover; single ops system covers all table types.

Conclusion

Apache Fluss's dual-table model redefines real-time data storage paradigms: LogTable uses columnar streaming storage to solve high-throughput log transport; PrimaryKeyTable uses integrated log+KV architecture to solve real-time business state updates. Both share underlying storage, cold/hot tiering, and lakehouse sinking, completely breaking the traditional fragmentation of "queue, cache, lakehouse, OLAP" into multiple components, providing a solid storage foundation for building lightweight, integrated real-time lakehouse platforms. This table organization system, combining performance, flexibility, and scalability, is the core differentiator of Fluss from traditional message queues and pure lakehouse formats, and the key cornerstone for its role as next-generation real-time data infrastructure.

Reference: Apache Fluss official website: fluss.apache.org