How Hologres Achieves Ultra‑High‑Throughput Real‑Time Writes in Big Data Warehouses

Challenges of Real-Time Data Ingestion: Performance, Updatability, Scale

In big‑data scenarios, writing real‑time data into a real‑time data warehouse is a major topic. Common write patterns include:

Append‑only: Traditional log‑type data (logs, event tracking) where records are independent and can simply be appended.

Insert or Replace: Based on a primary key (PK), if the PK does not exist the record is appended; if it exists, the existing record is fully overwritten. Typical use cases are:

Upstream database binlog synchronization.

Flink results continuously written, requiring an Insert‑or‑Replace target table.

Lambda architecture offline back‑fill, where yesterday's records are refreshed in the real‑time result table.

Insert or Update: Multiple applications update different fields of the same row, effectively joining multiple data sources. If the row exists, each application updates its fields via PK; if not, the first writer inserts the row. Typical scenarios:

User‑profile applications for real‑time risk control or ad targeting, where several Flink jobs write different dimensions to the same row.

Real‑time and offline data integration, storing both real‑time and offline results of the same PK in different columns for easy joint query.

In the following sections, Insert‑or‑Replace and Insert‑or‑Update are collectively referred to as Upsert .

Key Challenges for Efficient Real‑Time Writes

Challenge 1: Merge‑on‑Read vs. Merge‑on‑Write? In Upsert mode, the merge of new and old data can happen at write time (Merge‑on‑Write) for better query performance, or at read time (Merge‑on‑Read) for better write performance. Merge‑on‑Read harms query speed because each query must reconcile multiple versions.

Example of Merge‑on‑Read:

Example of Merge‑on‑Write:

Challenge 2: Primary‑Key Model Support? Supporting a primary‑key model is crucial for Upsert. Without PK, updates degrade to full‑table updates and Merge‑on‑Read cannot be applied.

Challenge 3: Exactly‑Once Guarantees? The system must ensure that, despite upstream failover causing duplicate writes, only a single latest record exists (either via Merge‑on‑Write or Merge‑on‑Read).

Challenge 4: Write‑Visibility Latency? Some workloads (BI) tolerate seconds‑level delay, but real‑time risk control or dashboards require data to be visible immediately after write.

Challenge 5: Massive Scale and Ultra‑High RPS? When RPS reaches hundreds of thousands or millions and table size grows to billions of rows, the system must still sustain high‑performance Upserts.

Hologres Real‑Time Write Model and Performance

Hologres, Alibaba Cloud’s self‑developed one‑stop real‑time data warehouse, was designed with the following capabilities:

Primary‑key support for efficient updates and deletions.

Full Upsert support: high‑performance Append‑Only, Insert‑or‑Replace, and Insert‑or‑Update.

Automatic Merge‑on‑Write for columnar tables and Merge‑on‑Read for row‑based tables.

For columnar tables, complex OLAP queries prioritize read performance, so Merge‑on‑Write is used. For row‑based tables, point‑lookup queries dominate, allowing Merge‑on‑Read to favor write throughput.

Exactly‑Once semantics via per‑row SQL transactions and primary‑key deduplication. Whether batch‑loading billions of rows or single‑row real‑time writes, each SQL statement is atomic (ACID). Upserts automatically overwrite existing rows; Append‑only writes ignore duplicates.

Write‑and‑visible instantly. Hologres has no build or batch‑flush phase; a successful SQL write returns immediately, making the data queryable without delay.

These five design choices mirror traditional databases but are applied at big‑data scale, enabling ultra‑high RPS writes and petabyte‑level storage.

Benchmark on a Hologres 128C instance with 10 concurrent writers inserting into a 20‑column columnar table:

Case 1: Table without primary key.

Case 2: Table with primary key (Insert‑or‑Replace) where new PK never conflicts.

Case 3: Table with primary key (Insert‑or‑Replace) where every insert conflicts; table size 200 M rows.

Case 4: Same as Case 3 but table size 2 B rows.

Result interpretation:

Case 2 shows only a small performance penalty for primary‑key existence checks compared to Case 1.

Cases 3 and 4 demonstrate that Hologres can locate and mark deletions with minimal overhead even when PK conflicts occur on tables with billions of rows.

Comparison with Common Products

Key differences among several data‑warehouse solutions:

ClickHouse : Batch writes; Merge‑on‑Read; duplicate PKs appear in query results until compaction.

Doris : Same as ClickHouse; Merge‑on‑Read causes query‑time merging overhead.

Data‑lake engines (Hudi, Iceberg, Delta Lake) : Batch writes; Merge‑on‑Read or Copy‑on‑Write; often cause full‑table rewrites and I/O amplification.

Hologres : Streaming writes with immediate visibility; Merge‑on‑Write; strong PK model makes updates/deletes cheap; delete‑bitmap technique ensures updates do not affect query performance.

Merge‑on‑Write Upsert Implementation

Typical Upsert (Insert‑or‑Replace) SQL:

CREATE TABLE users (
    id int NOT NULL,
    name text NOT NULL,
    age int,
    PRIMARY KEY(id)
);
INSERT INTO users VALUES (?,?,?)
ON CONFLICT(id) DO UPDATE
SET name = EXCLUDED.name,
    age = EXCLUDED.age;

In a pure Merge‑on‑Read approach, new files are appended and queries must join multiple versions, which severely degrades query speed.

Under Merge‑on‑Write, Upsert consists of three steps:

Locate the file containing the old record.

Process the old record (mark delete, etc.).

Write the new record.

High‑RPS Upsert requires all three steps to be fast.

1. Locating Old Data Files

Techniques include:

Bloom filter : Per‑file bloom filter quickly excludes files that do not contain the key.

Range filter : Stores min/max values per file; effective when keys are roughly increasing.

External index : E.g., HBase index mapping PK to file ID, though it introduces consistency and performance constraints.

2. Processing Old Data + Writing New Data

Two common methods:

Copy‑on‑Write : Merge old file with new data to produce a new file, causing write amplification.

Delta files : Mark old rows as deleted in a delta file and append new rows; queries must join delta files but write amplification is avoided.

Hologres Memtable‑Based Write Mechanism

Hologres follows Merge‑on‑Write using a strong primary‑key model, a memtable + WAL architecture, enabling high‑frequency writes.

1. File Model

Each columnar table maintains three file types:

Primary‑key index file (row‑oriented) storing PK → unique_id and clustering index.

Data file (columnar) with clustering index + unique_id sparse index and range filter.

Delete‑bitmap file: one bitmap per file; a bit set to 1 marks the corresponding row as deleted, providing near‑zero query impact.

All files are first written to a memtable; once the memtable reaches a threshold it becomes immutable and is flushed to disk by asynchronous threads.

2. Upsert Flow

When PK does not conflict, an Upsert costs one index lookup plus two memory writes. When PK conflicts, it costs an index lookup, file/row location, and three memory writes.

3. Upsert Example

Assume a table users with PK id and columns name, age:

CREATE TABLE users (
    id text NOT NULL,
    name text NOT NULL,
    age int,
    PRIMARY KEY(id)
);

Initial data:

u0, 张三, 10

u1, 李四, 11

u2, 王五, 12

Execute an Upsert that updates u1:

INSERT INTO users VALUES ('u1','新李四',12)
ON CONFLICT(id) DO UPDATE
SET name = EXCLUDED.name,
    age = EXCLUDED.age;

After the operation, the table becomes:

u0, 张三, 10

u1, 新李四, 12

u2, 王五, 12

Full‑Link Write Optimizations in Hologres

Hologres is fully PostgreSQL‑compatible (syntax, semantics, protocol), allowing direct use of PostgreSQL JDBC drivers.

1. Fixed Plan – Reducing Optimizer Overhead

For recognized Upsert patterns, the query optimizer applies a shortcut: the Upsert bypasses the full optimizer pipeline and is handled by a lightweight Fixed Planner that builds a Fixed Physical Plan directly from the AST.

2. High‑Performance Internal Communication

Reactor model with lock‑free asynchronous processing per shard.

Custom binary‑row protocol minimizes memory allocation and copying.

Back‑pressure and batch‑coalescing in the client improve throughput without adding latency.

3. Stable and Reliable Backend

Implemented in pure C++ for maximum performance.

Extensive caching layers (row, block, iterator, meta) reduce I/O and accelerate hot‑data access.

Concurrent I/O merging and dictionary‑based compression lower storage cost and I/O volume.

Conclusion

Hologres, developed by Alibaba, is a one‑stop real‑time data warehouse engine that supports massive‑scale real‑time ingestion, updates, and analytics with standard SQL (PostgreSQL‑compatible). It handles petabyte‑scale OLAP and ad‑hoc queries, delivers low‑latency online services, and has demonstrated write peaks of over 1.1 billion rows per second in Alibaba’s Double‑11 events.

Unlike many data‑warehouse products that sacrifice read or write performance, Hologres leverages a memtable architecture, delete‑bitmap technique, and primary‑key model to achieve balanced high‑throughput reads and writes. Integrated natively with Flink, Spark, and other compute engines, it supports a variety of streaming and batch scenarios.

As Hologres continues to evolve, further deep‑dives into its storage engine, query engine, and high‑throughput write mechanisms will be released.