Building a Lakehouse on Alibaba Cloud AnalyticDB (ADB) with Apache Hudi: Architecture, Challenges, and Practices

Li Shaofeng, a database expert from Alibaba Cloud, introduces the ADB Lakehouse edition, which unifies data collection, storage, computation, management, and application layers, offering low‑cost, high‑throughput storage, elastic resources, and a unified metadata service.

The architecture provides five upgraded capabilities: a one‑click data pipeline (APS) for easy ingestion, Hudi‑based storage supporting both offline and online workloads, an enhanced XIHE/BSP SQL engine plus Spark for complex processing, a unified metadata and permission service, and AI‑enabled analytics via Spark.

Key advantages include resource and experience integration, enabling a single data copy to serve both batch and real‑time scenarios with elastic scaling and cost reduction.

When building a lakehouse with Hudi, the team faced challenges such as 4 GB/s ingestion throughput, severe data skew, massive scan volumes, and limited elasticity of traditional warehouses.

The solution architecture uses SLS as the source, Flink for real‑time processing, and Hudi for storage, with a coordinated commit protocol ensuring exactly‑once semantics and automatic fallback between MPP and BSP execution modes.

To address hotspot and OOM issues, a hotspot‑shuffling mechanism, memory‑efficient Parquet writing, and OSS request optimizations (timeline‑based checkpoint metadata, SDK tweaks, and marker‑file caching) were implemented.

Concurrent write support was achieved by isolating checkpoint metadata, view storage, and table service instances, adding retry strategies for instant generation, and ensuring table‑service isolation.

Lifecycle management features allow partition‑level data retention based on size, count, or expiration time, supporting concurrent policy updates.

The independent TableService handles background compaction, commit cleanup, and asynchronous clustering, improving query performance by over 40% while keeping table state size under control.

In practice, the ADB Lakehouse combines APS for low‑cost, low‑latency ingestion, Spark for batch and ML workloads, and zero‑ETL integration that lets users query ADB tables directly from Spark without data movement.

Overall, the ADB Lakehouse provides a cloud‑native, one‑stop data analysis platform that bridges the gap between data lakes and warehouses, delivering high performance, elasticity, and cost efficiency.