Alibaba Cloud Native Data Lake with Apache Iceberg: Architecture, Challenges, and Solutions

This talk, presented by Alibaba Cloud technical expert Hu Zheng and edited by Zhang Detong of DataFunTalk, introduces the concept of a cloud‑native data lake and compares it with traditional databases and data warehouses.

It explains the differences among databases, data warehouses, and data lakes, highlighting that data lakes store raw data in columnar files, have lower storage costs, and support diverse compute models such as SQL, graph, and machine‑learning workloads.

The speaker then discusses the challenges of moving workloads to the cloud, including the need for elastic storage, the high cost of maintaining on‑premises HDFS, and the benefits of object storage (OSS/S3) with built‑in optimizations.

Using Hive as an example, the presentation lists specific difficulties when migrating to the cloud: lack of ACID guarantees, difficulty replacing HDFS with S3, schema incompatibilities, and metadata scalability issues.

Key challenges for cloud data lakes are identified: a unified metadata center, real‑time multi‑source ingestion, and enterprise‑grade data security and isolation.

The core of the solution is Alibaba Cloud's Iceberg‑based data lake. Iceberg’s table format provides a storage‑agnostic, open‑source standard with three layers—files, manifest lists, and snapshots—enabling ACID transactions, time‑travel queries, and efficient reads.

Iceberg’s architecture is illustrated with a series of snapshot diagrams showing how each transaction creates new data files and manifests while preserving previous snapshots for consistent reads.

To address metadata challenges, Alibaba Cloud offers Data Lake Formation (DLF) as a unified catalog that manages database, table, and location mappings as well as schema and partition information, enabling data lineage tracking and fine‑grained access control.

For real‑time ingestion, Flink is recommended to write data into OSS using Iceberg tables, providing exactly‑once semantics and isolating write workloads from analytical queries.

The solution also automates schema evolution: Flink captures DDL changes from upstream sources and propagates them to Iceberg, allowing instant schema synchronization without manual intervention.

To handle small‑file proliferation, three strategies are described: bucket‑based shuffling during Flink writes, periodic batch compaction, and incremental streaming compaction for low‑volume updates.

CDC use‑cases are covered by showing how MySQL change streams can be ingested into Iceberg via Flink, with subsequent analysis across Spark, Hive, or Presto, while noting the need for occasional compaction.

A comprehensive testing framework validates end‑to‑end data correctness across failure scenarios (e.g., task manager restarts, storage outages, high CPU load), ensuring reliability of both ingestion and query pipelines.

The summary highlights four main advantages of Alibaba Cloud’s Iceberg data lake: open data formats, diverse compute engine support, elastic resource scaling, and professional support from dedicated product teams.

The session concludes with a thank‑you note and a call for audience engagement.