Apache Fluss: Four Merge Engines Explained for Real-Time Analytics

Apache Fluss's merge engines run on the write path of primary-key tables, determining how multiple records with the same key are merged. This enables a single table to serve different business semantics—latest state, first occurrence, versioned consistency, or pre-aggregated metrics—by switching the merge engine parameter without changing the storage architecture.

1. Default Merge Engine (LastRow): General-Purpose Default

Core Principle

For each primary key, the engine always retains the last written full record. New writes directly overwrite the old version, behaving exactly like a database primary-key upsert.

Supported Operations

Full write overwrite : same-key full-field writes, later version overwrites earlier.

UPDATE SQL : supports standard UPDATE ... SET ... WHERE ... syntax (batch mode only).

DELETE SQL : supports standard DELETE FROM ... WHERE ... syntax.

Partial update : can write only the primary key and target columns; unspecified columns retain existing values—this is the engine's key enhancement.

Typical Behavior Example

CREATE TABLE T (
  k INT,
  v1 DOUBLE,
  v2 STRING,
  PRIMARY KEY (k) NOT ENFORCED
);
-- Same-key overwrite
INSERT INTO T VALUES (1, 1.0, 't1');
INSERT INTO T VALUES (1, 1.0, 't2');
-- Result: k=1, v1=1.0, v2='t2' (last row kept)
-- Partial column update
INSERT INTO T(k, v1) VALUES (3, 3.0);
INSERT INTO T(k, v2) VALUES (3, 't3');
-- Result: k=3, v1=3.0, v2='t3' (columns merged separately)

Applicable Scenarios

CDC synchronization from business databases, preserving latest business state.

Real-time orders, user profiles, product dimensions—general business state tables.

Real-time wide-table scenarios where multiple streams asynchronously update different fields.

Standard primary-key state storage with no special merge requirements.

2. FirstRow Merge Engine: First-Write Retention

Core Principle

Each primary key retains only the first arriving record; all subsequent same-key writes are ignored. The produced changelog is append-only (no retract events), so downstream can treat it as a log table.

Supported Operations and Limitations

Only INSERT writes are supported.

UPDATE and DELETE SQL are not supported.

Partial column updates are not supported.

Automatically ignores UPDATE_BEFORE and DELETE change events.

Typical Behavior Example

CREATE TABLE T (
  k INT,
  v1 DOUBLE,
  v2 STRING,
  PRIMARY KEY (k) NOT ENFORCED
) WITH (
  'table.merge-engine' = 'first_row'
);
INSERT INTO T VALUES (1, 2.0, 't1');
INSERT INTO T VALUES (1, 3.0, 't2'); -- ignored
-- Result: k=1, v1=2.0, v2='t1' (first row kept)

Applicable Scenarios

Streaming data deduplication, replacing Flink's Deduplicate operator.

First-user-behavior, first-order, first-visit snapshot statistics.

Downstream operators that don't support retract (e.g., window aggregation, interval join) and require a pure append stream.

3. Versioned Merge Engine: Version-Controlled Merging

Core Principle

User specifies a version field (numeric or timestamp). During merge, the engine compares version values: update only when new version ≥ stored version ; if new version is smaller or null, the write is ignored. This guarantees the highest-version record is kept regardless of write order.

Version Field Supported Types

INT

, BIGINT, TIMESTAMP, TIMESTAMP_LTZ and their precision variants.

Supported Operations and Limitations

Only INSERT writes supported.

UPDATE and DELETE SQL not supported.

Partial column updates not supported.

Automatically ignores UPDATE_BEFORE and DELETE change events.

Typical Behavior Example

CREATE TABLE VERSIONED (
  a INT,
  b STRING,
  ts BIGINT,
  PRIMARY KEY (a) NOT ENFORCED
) WITH (
  'table.merge-engine' = 'versioned',
  'table.merge-engine.versioned.ver-column' = 'ts'
);
INSERT INTO VERSIONED VALUES (1, 'v1', 1000);
INSERT INTO VERSIONED VALUES (1, 'v2', 999);  -- version smaller, ignored
-- Result: a=1, b='v1', ts=1000
INSERT INTO VERSIONED VALUES (1, 'v3', 2000); -- version larger, updated
-- Result: a=1, b='v3', ts=2000
INSERT INTO VERSIONED VALUES (1, 'v4', null);  -- version null, ignored
-- Result: remains a=1, b='v3', ts=2000

Applicable Scenarios

Out-of-order data streams deduplication and merging, ensuring eventual consistency.

Business data sync with version numbers, preventing stale data from overwriting fresh data.

Replacing Flink versioned deduplication logic by pushing merge down to storage layer.

4. Aggregation Merge Engine: Real-Time Pre-Aggregation

Core Principle

Each non-primary-key field can be configured with an independent aggregation function. On same-key writes, the engine aggregates field-by-field per function, storing the aggregated result. Fields without an explicit function default to last_value_ignore_nulls.

Built-in Aggregation Functions

Delete Behavior

Configured via table.delete.behavior: ignore (default): delete operations silently ignored. disable: reject delete operations with an error. allow: allow deletes—full-row delete in full-update mode, set target columns to null in partial-update mode.

Note: Retract semantics (e.g., sum decrement, max rollback) are not currently supported; deletes can only remove the whole row or nullify columns.

Typical Behavior Example

CREATE TABLE product_stats (
  product_id BIGINT,
  price DOUBLE,
  sales BIGINT,
  last_update_time TIMESTAMP(3),
  PRIMARY KEY (product_id) NOT ENFORCED
) WITH (
  'table.merge-engine' = 'aggregation',
  'fields.price.agg' = 'max',
  'fields.sales.agg' = 'sum'
);
INSERT INTO product_stats VALUES (1, 23.0, 15, '2024-01-01 10:00:00');
INSERT INTO product_stats VALUES (1, 30.2, 20, '2024-01-01 11:00:00');
-- Result: product_id=1, price=30.2 (max), sales=35 (sum), last_update_time=latest value

Applicable Scenarios

Real-time metric pre-aggregation: cumulative, max, min statistics.

Real-time dashboard and monitoring metric sink.

RoaringBitmap bitmap UV deduplication counting.

Real-time cumulative calculation of counters and gauge metrics.

3. Horizontal Comparison of Four Merge Engines

4. Production Selection Decision Guide

5. Production Landing Considerations

Engine selection must be decided upfront : merge engine is specified at table creation; switching later is discouraged to avoid semantic confusion.

Exactly-once guarantee : combined with Flink Checkpoint, all four engines achieve end-to-end exactly-once writes; native clients must implement their own undo rollback logic.

Delete semantics need careful evaluation : Aggregation engine does not support retract subtraction; delete scenarios involving cumulative metrics require upfront business impact assessment.

Partial updates only available in Default : when multi-stream merging requires partial column updates, Default is the only choice.

Prefer non-Default for pure append streams : if downstream doesn't support retract, choose FirstRow or Versioned to avoid correctness issues from retract events.

Conclusion: Storage-Layer Intelligent Merging, a New Paradigm for Real-Time Data Warehousing

Apache Fluss's four merge engines essentially push common real-time computing logic—merge, deduplication, version control, aggregation—down to the storage layer for native implementation. This turns primary-key tables from passive state containers into intelligent storage units that can flexibly switch behavior based on business semantics.

In practice, Default covers 80% of general scenarios and is the default optimal choice; FirstRow and Versioned specifically solve deduplication and out-of-order problems; Aggregation targets pre-aggregation scenarios and is a powerful tool for boosting real-time metric performance. Understanding the semantic boundaries and applicable scenarios of these four engines is a key step to mastering Fluss primary-key tables and building high-performance real-time data warehouses.