Building an Enterprise Real-Time Data Warehouse with Hologres and Flink at Cao Cao Mobility

Business Background Cao Cao Mobility, founded on May 21, 2015, is a strategic investment of Geely Holding Group focusing on a green shared‑mobility ecosystem. It provides ride‑hailing, car‑pooling, and dedicated‑car services, where users place orders on the platform, drivers receive dispatches, and payments are settled after service completion.

Business Pain Points The end‑to‑end data flow passes through marketing, order, dispatch, risk control, payment, and fulfillment systems, feeding into an RDS database and then a real‑time data warehouse for analytics, dashboards, BI, monitoring, and algorithmic decisions. The traditional Lambda architecture suffers from high component complexity, elevated R&D and operational costs, low operational efficiency due to Kafka‑based real‑time layers, and resource‑intensive large‑state Flink jobs.

Key Requirements Developers need a unified component set for diverse scenarios, reliable data correction in complex pipelines, and solutions for large‑state Flink processing.

Hologres Capability Analysis

Business Scenario Support OLAP analysis, high‑concurrency point queries, semi‑structured log analysis, and PostGIS‑based spatial queries.

One‑Stop Real‑Time Development Alignment with data‑warehouse layering, integrated Flink CDC and catalog, and ad‑hoc query capability across offline and real‑time data.

Pain‑Point Solutions Low‑latency full‑link, multi‑stream join support, and precise distinct‑count handling.

Hologres High‑Concurrency Update Support

Hologres uses a distributed storage system with a block‑cache engine, sharding into tablets and WAL. It adopts Merge‑On‑Read for row storage and Merge‑On‑Write for column storage, handling data files, delete markers, and auxiliary indexes (e.g., RoaringBitmap) before flushing immutable Memtables to data files.

Hologres Binlog Support

The Binlog stores physical tables with built‑in sequence, modification type, and timestamp, providing strong consistency with the original table and mapping Flink RowKind types for precise change capture.

Hologres Data Model Overview

Row storage is used for KV and subscription scenarios (high‑concurrency lookups, CDC), while column storage serves aggregation scenarios (OLAP, complex queries) with rich indexes such as clustered, bitmap, dictionary, and time‑range indexes. Log scenarios leverage JSONB for semi‑structured data.

Real‑Time Warehouse Architecture Design

Data flows from RDS via Binlog to Kafka ODS, then Flink writes to Hologres DIM, aggregates to DWD, and finally to DWS through another Binlog subscription. A unified query service (One Service) exposes the results to downstream applications.

DWD Wide‑Table Construction Practice

Wide tables leverage Hologres' column‑update capability for efficient joins. Dimension tables are built with start‑time/end‑time validity intervals, and asynchronous pipelines handle dimension latency, missing records, and periodic consistency checks.

Aggregation Computation Optimization

Two improvements were made: a custom MapSumAgg operator that sums map‑structured values, and dynamic configuration of Grouping Sets to add dimensions without restarting jobs, enabling flexible metric addition while preserving operator state.

Throughput Tuning of the Link

Write‑side: a Union layer performs PK‑based partitioning and small‑window pre‑aggregation before Flink writes to Hologres, reducing write pressure. Read‑side: lag‑window analysis on Binlog changes filters redundant updates, easing downstream load.

Metadata Lineage Refactoring

Integrating Flink Catalog with Hologres and Kafka enables multi‑version schema management. Kafka source/sink headers carry version info, allowing downstream tasks to detect dependencies and facilitate safe decommissioning.

Link Assurance System

Health monitoring collects Flink metrics and Kafka offsets, detecting back‑pressure, skew, checkpoint failures, and node‑level resource bottlenecks. Latency monitoring combines Kafka timestamps with Flink offsets to compute end‑to‑end delays, enabling rapid pinpointing of anomalies.

Data Correction Capability Construction

The correction workflow copies the original Flink job, creates a temporary Hologres table, redirects the sink, and performs a stateless replay. After completion, a script reconciles the original and corrected tables, ensuring downstream transparency and minimal disruption.

Business Outcome Analysis

The Hologres+Flink architecture reduces component count, lowers development complexity, shortens delivery cycles, improves data reusability, simplifies operational monitoring, and cuts storage and compute costs by consolidating offline and real‑time layers.

Future Outlook

Plans include dynamic scaling of self‑built Flink clusters, intelligent resource auto‑tuning, broader adoption of Flink CDC for unified ODS ingestion, and leveraging Hologres master‑slave isolation to support multiple data services with high availability.