Unlock Real‑Time Analytics with Hologres Dynamic Table: A Quick‑Start Guide

Hologres Dynamic Table Introduction

Typical real‑time data‑warehouse scenarios include BI reporting, user‑operation analysis, log analysis, and monitoring, which are widely used in advertising, gaming, e‑commerce, and internet industries. Not all scenarios require sub‑second latency; many are quasi‑real‑time with minute‑level delay.

The common architecture consists of offline and real‑time storage layers, with compute engines such as MaxCompute, Spark for batch processing and Flink for streaming, forming a Lambda architecture. This setup introduces complexity due to multiple components, data redundancy, and inconsistent data semantics between offline and online layers.

Dynamic Table, introduced in Hologres 3.0, offers a new table type that materializes query results automatically and supports two computation modes—incremental refresh and full refresh—to address these challenges.

Dynamic Table Usage

Specify the refresh mode (incremental or full) when creating a Dynamic Table.

Set the refresh interval to control how frequently the table updates.

Define the query that the Dynamic Table will execute and store the results.

Query the Dynamic Table directly to obtain the materialized results.

Advantages of Dynamic Table

Declarative data processing: automatic refresh without external scheduling scripts, simplifying operations.

Very low‑latency updates: incremental refresh can achieve minute‑level latency, suitable for quasi‑real‑time scenarios.

Standard SQL‑based development: no extra learning cost, supports common SQL features such as JOIN, GROUP BY, and window functions.

Unified multi‑mode computation: switch between incremental and full refresh by changing a single parameter.

Dynamic Table Demo

The demo covers three scenarios:

Creating a full‑refresh Dynamic Table (end‑to‑end).

Creating an incremental‑refresh Dynamic Table based on a Parquet external table.

Creating a hybrid Dynamic Table that combines full and incremental refreshes for large tables with partitioned data.

Usage Recommendations

Prefer incremental refresh when possible, as it processes less data and consumes fewer resources, though it has some limitations and requires enabling the base table's log.

Monitor refresh latency using the system table Dynamic_Table_Refresh_History and its Refresh_Latency field.

Use partitioned tables: apply incremental refresh to recent partitions and full refresh to historical partitions, enabling a single codebase to handle multiple refresh strategies.

Choose between local instance and Serverless resources based on task size; Serverless is recommended for large or historical full‑refresh jobs.

Set up monitoring and alerting for refresh tasks (e.g., QPS, RPS, failures) via cloud monitoring services.

Comparison with Other Products

Dynamic Table is comparable to Snowflake’s Dynamic Tables, both supporting batch and minute‑level incremental refreshes, multi‑table joins, and rich observability. Compared with DIS asynchronous materialized views and Doris’s materialized views, Hologres Dynamic Table offers true incremental refresh, reducing processing volume and resource consumption for minute‑level latency scenarios.