CaoCao Mobility's Real‑Time Data Warehouse: Hologres + Flink

Business Background

CaoCao Mobility, founded on May 21, 2015, is a strategic investment of Geely Holding Group aimed at reshaping green shared mobility using internet, vehicle‑networking, autonomous driving, and new‑energy technologies. The platform offers ride‑hailing, car‑pooling, and dedicated‑car services, with order placement, driver dispatch, fulfillment, and payment forming the core workflow.

Business Pain Points

Data from marketing, orders, dispatch, risk control, payment, and fulfillment flows into an RDS database and then into a real‑time data warehouse for tagging, dashboards, BI, monitoring, and algorithmic decisions. The traditional Lambda architecture suffers from high component count, elevated development and operational costs, low operational efficiency due to Kafka‑based pipelines, and resource‑intensive large‑state Flink jobs.

Complex component ecosystem.

High R&D effort for both real‑time and batch pipelines.

Low operational efficiency and difficult data debugging.

Significant resource consumption for large‑state processing.

Developers also require a unified component set, reliable data correction, and solutions for large‑state challenges in Flink.

Hologres + Flink Enterprise Real‑Time Warehouse

Hologres Capability Overview

Hologres provides rich OLAP analysis, high‑concurrency point queries, semi‑structured log analysis, and PostGIS‑based spatial extensions, making it suitable for CaoCao’s business scenarios.

One‑Stop Real‑Time Development

Aligns with data‑warehouse layering concepts, enabling real‑time data flow and storage.

Deep integration of Flink CDC and Flink Catalog.

Unified ad‑hoc capability for accelerated federated analysis.

Key Pain Points Solved

End‑to‑end low latency.

Multi‑stream join support for primary‑key and row‑level updates.

Accurate distinct‑count handling for large state.

Real‑Time Warehouse Architecture Design

Data originates from RDS, is captured via Binlog into Kafka ODS, then Flink writes to Hologres DIM. Flink aggregates ODS streams into Hologres DWD, where wide tables are built. Subsequent Binlog‑driven ingestion populates Hologres DWS, which is exposed through a unified query service.

DWD Wide‑Table Construction

Leveraging Hologres’s column‑update capability, the team builds wide tables that support both slowly changing dimensions and frequently changing ones, handling dimension latency with compensation mechanisms and periodic consistency checks.

Aggregation Computation Optimizations

Two main improvements were made:

Implemented a MapSumAgg operator that extends SumAgg to support map‑based aggregation.

Enabled dynamic configuration of Grouping Sets to add dimension granularity without restarting jobs.

These changes reduce state explosion and improve reuse of aggregation logic.

Throughput Tuning

Write‑side: a Union layer performs pre‑aggregation on PK‑partitioned streams before writing to Hologres, reducing write pressure.

Read‑side: lag‑window techniques filter redundant updates from Binlog, decreasing downstream load.

Metadata Lineage Refactoring

Integrating Flink Catalog with Hologres and custom Kafka Topic catalogs enables schema evolution, versioned data tracking, and streamlined release pipelines.

Link Assurance System

Monitoring collects Flink metrics and Kafka offsets, detecting anomalies such as back‑pressure, skew, checkpoint failures, and node resource saturation, providing rapid root‑cause diagnosis.

Data Correction Capability

A three‑step process uses Hologres to create a temporary correction table, redirects Flink sinks to it, and performs stateless replay, ensuring corrected data is merged without disrupting downstream consumers.

Business Outcomes

Clearer architecture with fewer components compared to Lambda.

Reduced development complexity and faster delivery cycles.

Improved operational experience with easier data inspection and correction.

Lowered component maintenance and storage costs by consolidating real‑time and offline stores.

Reduced large‑state processing overhead in Flink.

Future Outlook

Enhance Flink cluster elasticity for peak‑load scaling and seamless down‑scaling.

Introduce dynamic resource perception and intelligent auto‑tuning.

Adopt Flink CDC as a unified ODS ingestion method, addressing encryption and archival needs.

Leverage Hologres master‑slave isolation to support multi‑tenant data services.