How Hologres Transformed a Real‑Time Data Warehouse: Cutting Costs & Boosting Performance

1 Customer Overview

Tal Education (NYSE: TAL) is a technology‑driven education company offering smart education services worldwide, covering public and private sectors from age -1 to 24. Its online school brand, Xueersi Online School, serves children aged 6‑14 with a dual‑teacher live‑stream model and AI‑enhanced courses.

2 Background of Real‑Time Data Warehouse Development

Real‑time Warehouse 1.0 was built in 2019 on the Kudu OLAP engine. Initially stable with low load, rapid business growth after 2020 increased data volume and task count, exposing Kudu’s performance limits, high operational cost, and maintenance difficulty.

Following the 2021 "double reduction" policy, the company faced shrinking business and wasted resources, prompting a cost‑governance initiative. After evaluating several OLAP engines, Hologres was selected, and the upgrade to Warehouse 2.0 began in January 2022, delivering lower cost and higher reliability.

3 Real‑Time Warehouse 1.0: Kudu‑Based OLAP Engine Shows Bottlenecks

3.1 Architecture Overview

The warehouse supports reporting, precise marketing, and other scenarios through four layers: ODS, DWD, DWS, and ADS.

ODS stores raw logs and business data synchronized from source systems.

DWD cleans ODS data and partitions it by business domain (teaching, transaction, marketing, etc.).

DWS joins DWD with dimension tables to generate wide tables or aggregated models.

ADS provides data to applications via MySQL or PolarDB for dashboards, real‑time interfaces, and other services.

3.2 Kudu‑Based Scenario Solutions

Two real‑time models were built:

Minute‑level (near‑real‑time) model using Spark/Flink to preprocess data before writing to Kudu, then moving results to PolarDB/MySQL for dashboards and reports.

Second‑level (real‑time) model using Flink + Kafka, writing DWD details to Kudu, aggregating in DWS, and outputting to ADS via PolarDB, MySQL, or Kafka for online services.

Approximately 80% of scenarios use the minute‑level pipeline, while 20% require second‑level processing.

3.3 Business Challenges with Kudu

Impala nodes suffer memory pressure under heavy concurrent SQL tasks, causing failures.

Lack of specialized Kudu operations staff leads to long troubleshooting cycles and unstable SLA.

Node failures require rapid scaling, increasing operational and cost pressure.

The "double reduction" policy forces urgent cost reduction, making Kudu replacement essential.

These issues motivated the search for a more cost‑effective, stable OLAP solution.

4 Real‑Time Warehouse 2.0: Hologres Replaces Kudu

4.1 OLAP Engine Selection Criteria

Strong OLAP capabilities

SQL support with update, delete, upsert

High throughput and high availability

Easy operations and elastic resource scaling

After comparing market options, Hologres was chosen as the primary engine.

4.2 Hologres Fully Replaces Kudu

All data pipelines now use Hologres; a read‑only replica replaces the previous PolarDB/MySQL query engines, achieving read‑write separation.

Offline data is ingested via the T‑Collect tool into Hologres ODS; real‑time data arrives through Flink from MySQL binlog and logs.

Four layers (ODS, DWD, DWS, ADS) are maintained in Hologres, scheduled and cleaned by the T‑Data platform, with the replica serving online queries.

Both offline and real‑time data are stored in Hologres, supporting dashboards, large screens, APIs, and push services.

4.3 Unified Query Engine on Hologres Replica

In Warehouse 1.0, results were written from Kudu to PolarDB/MySQL, creating a long data‑movement chain. Warehouse 2.0 adopts Hologres shared‑storage multi‑instance high‑availability, with the primary handling load and the replica serving all queries, eliminating the ADS sync step and reducing development cost.

Future plans include exposing the Hologres replica to analysts via the T‑Query tool for ad‑hoc exploration, further lowering manual effort.

5 Benefits: Cost Reduction and Efficiency Gains

5.1 Million‑row Writes and Millisecond Queries

Hologres can ingest over a million rows per second and deliver sub‑second OLAP queries, accelerating data‑driven decisions.

Multi‑instance deployment isolates read and write workloads, ensuring stable performance.

5.2 Near‑Million‑Dollar Annual Cost Savings

Consolidating to a single Hologres system eliminates the need for multiple OLAP stacks, saving close to one million USD per year.

Task and Yarn queue optimizations cut resource costs by tens of thousands per month and reduce storage redundancy by 90%.

5.3 Reduced Operations Burden

Alibaba’s managed service lessens operational overhead, allowing the team to focus on business logic and data quality.

Elastic scaling handles peak loads (e.g., holidays) and downsizing during low‑traffic periods.

5.4 Enterprise‑Wide Promotion

The successful Warehouse 2.0 architecture is being replicated across the group to support core real‑time services such as renewal tracking, conversion, and enterprise WeChat integration.

6 Future Plans and Expectations

Continue building the real‑time warehouse, enhancing metadata, data quality, asset management, and security.

Explore stream‑batch unified processing techniques.

Open issues include the lack of user‑defined functions in Hologres, complex permission models that increase development effort, and the desire for native Hive table querying to simplify stream‑batch integration.

Overall, the team expects Hologres to evolve into a more feature‑rich, user‑friendly OLAP engine that further empowers real‑time data warehousing.