Boost Real‑Time Data Warehouses with Integrated Analytics & Service

Integrating analysis and service capabilities is a key innovation of Alibaba Cloud's unified data warehouse.

Decision‑making through data involves various analysis types—multidimensional, exploratory, interactive, ad‑hoc—typically used for internal reports, dashboards, and metric platforms, handling complex queries. Service workloads, often referred to as TP, support high‑performance, high‑QPS online data reads/writes with strict SLA, availability, and latency requirements, focusing more on throughput than strict transactionality, and serving 2C scenarios such as recommendation, marketing, and risk control.

Both analysis and service share the same underlying data sources and can support each other: service data can be re‑analyzed, and analytical data can power online services. An integrated architecture simplifies data exchange between systems and improves development efficiency.

A reliable, efficient real‑time data warehouse is becoming more agile, with lightweight, real‑time processing and weakened data layering. In service domains, big‑data teams are shifting from cost centers to profit centers, ensuring stable, high‑efficiency online operations. Architectural integration of analysis and service boosts development efficiency and reduces operational costs.

1. Traditional Lambda Architecture – Complexity and Pain Points

Historically, building a real‑time big‑data warehouse relied on the Lambda architecture, featuring separate real‑time, offline, and near‑real‑time processing layers, with distinct offline and online storage (OLAP and KV systems). APIs accessed online systems, while SQL accessed analytical systems, each tied to different storage engines.

This architecture works when business changes are minimal and data quality is high, but real‑world scenarios demand agile changes and often suffer from poor data quality. Frequent schema adjustments, data cleaning, and re‑processing become time‑consuming. Multiple storage systems and repeated data synchronization hinder agility, increase resource consumption, raise development costs, and make talent acquisition harder.

2. Data Processing Agility

The core of processing agility lies in two aspects: simplifying state storage to reduce data redundancy, allowing developers to modify data in a single place, and lightweight processing pipelines.

Hologres offers efficient real‑time batch writes and updates, supporting both single‑row flexible updates and massive batch rewrites, enabling a unified state layer that reduces data movement.

Processing is divided into a common layer and an application layer. The common layer uses Flink + Hologres Binlog to drive ODS→DWD→DWS in real time. The application layer encapsulates business logic, reduces intermediate tables, and leverages Hologres’s distributed query capabilities to give analysts flexibility while offloading engineering effort.

3. Data Service Online‑ization

Data is expanding from 2B internal decision‑making to 2C online business scenarios such as real‑time profiling, personalization, and risk control. Online services demand higher availability, concurrency, low latency, cloud‑native elasticity, hot upgrades, and robust observability.

1. Online‑driven Reliability Design

Hologres adds a hybrid row‑column storage model, allowing a single table to serve both OLAP and KV use cases, and introduces shard‑level multi‑replica for linear QPS scaling. This also enables non‑primary‑key point queries, useful for order lookups.

Hot upgrades keep services running during maintenance, while metadata physical backups and lazy data‑file opening accelerate fault recovery, achieving over ten‑fold speed improvements and minute‑level automatic recovery.

2. Integrated Analytics‑Service (HybridServing/AnalyticalProcessing, HSAP)

The integrated approach supports both complex OLAP analysis and high‑QPS, low‑latency online services within a single architecture, providing a unified data service endpoint, reducing data silos, and simplifying operations.

To meet diverse workloads, Hologres implements both row and column storage: row storage for high‑QPS online queries, column storage for OLAP, and fine‑grained load isolation on shared data.

3. Resource Isolation, High Availability, Unlimited Elasticity

Hologres offers multi‑instance, high‑availability shared storage. Multiple instances share the same data; one primary handles reads/writes, others are read‑only. Memory state syncs in milliseconds, providing full isolation of compute resources while keeping a single data copy.

In the 2021 Double‑11 peak, this design achieved near‑zero failures. Recommended practice: use one primary for data ingestion and processing, and allocate read‑only sub‑instances for analytical or external data services, matching compute resources to workload needs.

4. Case Study: Real‑Time Warehouse Upgrade for a Leading Logistics Company

The logistics firm required real‑time decision‑making and could face massive traffic spikes during promotions, demanding high‑performance online services.

Previously, they used traditional relational databases for real‑time queries and monitoring, which suffered from latency, complex joins, and instability under high concurrency, especially during Double‑11 traffic surges.

The solution replaced the legacy stack with Flink + Hologres. High‑frequency service data is streamed from DataHub via Flink into Hologres; analytical data is ingested from RDS binlog, built into ODS/DWD/DWS layers in Hologres, and exposed to applications for fast, concurrent queries.

This hybrid model leverages Flink’s stream processing and Hologres’s multidimensional query engine, eliminating traditional OLAP and RDS components, simplifying the architecture.

Post‑upgrade, system stability improved dramatically, achieving zero failures during Double‑11, supporting real‑time parcel tracking, intra‑warehouse operations, and delivering strong real‑time data support for operations.

The elastic cloud‑native capabilities of Hologres allowed dynamic scaling to handle traffic spikes thousands of times higher than normal, reducing operational costs.

Alibaba Cloud Big Data provides simple, fully managed, cloud‑native services that activate data productivity and generate business value.