Iceberg Data Lake: Core Features, Xiaomi Use Cases, and Future Plans

Overview

This talk covers the application scenarios of the Iceberg data lake at Xiaomi.

Main Content (Three Parts)

1. Iceberg Core Features

Iceberg is an open standard for tables with SQL behavior, supporting structured data schemas and offering open‑source design that works with multiple file formats, storage back‑ends (including cloud), and query engines such as Flink, Spark, Hive, and Trino. Key characteristics include transactional operations, full schema evolution, implicit partitioning, and row‑level updates using position‑delete and equality‑delete files.

1.1 Transactionality

All table operations are atomic, and multi‑snapshot reads provide read‑write separation.

1.2 Full Schema Evolution

Schema changes such as type promotion, column addition, deletion, renaming, and reordering are supported, though only type promotions (e.g., int to long) are allowed.

1.3 Implicit Partitioning

Iceberg offers various partition functions and multi‑level partitioning, automatically managing partition directories without user‑specified paths, thus avoiding partition drift and simplifying query planning.

1.4 Row‑Level Update Capability

Version 2 introduces row‑level updates via position‑delete and equality‑delete files; Flink implements this by using position deletes for new snapshots and equality deletes for older data, with Merge‑On‑Read queries delivering the final view.

2. Iceberg Use Cases at Xiaomi

2.1 Log Ingestion

Original pipeline: client SDK → Talos (Kafka‑like MQ) → Spark Streaming → HDFS → Hive. Issues: at‑least‑once semantics causing duplicates, lack of schema leading to partition drift, and schema mismatches between Hive and files.

New pipeline: MQ with schema → Flink SQL → Iceberg. Benefits: exactly‑once processing, implicit partitioning, and schema‑on‑write with evolution.

2.2 Near‑Real‑Time Data Warehouse (Flink + Iceberg)

Challenges with IoT device metrics: delayed uploads, batch Spark jobs causing resource spikes, and difficulty handling late data.

Solution: Flink SQL with exactly‑once semantics writes directly to Iceberg, using date + event_name as a secondary partition to reduce scan volume and spreading computation throughout the day via Flink checkpoints.

2.3 Offline Scenario Issues

Partition completeness checks, watermark‑based partition finalization, and optimization techniques such as local sort instead of costly Z‑order, and Parquet page‑column indexes for better predicate push‑down.

2.4 Column‑Level Data Encryption

Iceberg’s metadata layer enables column‑level encryption using Parquet 1.12.2. Xiaomi implements a single‑layer encryption where a DEK is stored in Iceberg metadata and encrypted/decrypted via a KeyCenter service, reducing KeyCenter load compared to per‑row encryption.

2.5 Hive‑to‑Iceberg Migration Research

Three approaches were evaluated:

① In‑place migration using Spark’s CALL migrate procedure – limited by supported file formats and Spark version requirements.

② Reusing Hive location – problematic due to Iceberg’s multi‑snapshot nature causing duplicate reads.

③ Creating new Iceberg tables and back‑filling historical data – chosen for its flexibility, despite migration effort, and yields ~30% storage savings via ZSTD compression and sorted large strings.

CALL catalog_name.db.sample('Spark_catalog.db.sample', map('foo','bar'))

set Spark.sql.sources.partitionOverwriteMode=static;

insert overwrite catalog.db.table_test partition(date=20230101) values (1,'c',1), (2,'d',2);

2.6 Real‑Time Integration (MySQL/TiDB/Oracle binlog → MQ → Flink → Iceberg v2)

Provides end‑to‑end exactly‑once semantics, minute‑level latency, and transactional guarantees, though compaction must be managed manually.

2.7 Current Iceberg Adoption at Xiaomi

Over 14,000 tables, daily table creation surpasses Hive, with total data volume exceeding 30 PB.

3. Future Plans

• Implement materialized view support for OLAP workloads without predicate push‑down.

• Follow Iceberg’s changelog view on Spark 3.3 for incremental reads.

• Explore cloud migration for overseas clusters to reduce EBS storage costs.

4. Q&A (selected)

Q1: Why replace Spark Streaming with Flink SQL?

A: Spark 2.3’s at‑least‑once semantics caused duplicates; Flink offers exactly‑once and aligns with Xiaomi’s strategic shift.

Q4: Why choose Iceberg over Hudi?

A: Iceberg’s openness, multi‑engine support, and early Flink integration made it a better fit for Xiaomi’s lambda/kappa architecture.

Q6: Latency for append and upsert modes?

A: Minimum checkpoint interval is 1 minute, achieving sub‑minute latency for both modes.

Thank you for attending.