Building a Real-Time Data Warehouse with Apache Flink and Apache Iceberg: Architecture, Challenges, and Best Practices

Apache Flink is a popular unified stream‑batch engine, and data lakes such as Iceberg, Hudi, and Delta have emerged to meet cloud‑era storage needs; Iceberg currently supports Flink DataStream/Table API writes and integrates with Flink 1.11.x.

The talk, delivered by senior engineer Su Shu from Tencent, outlines the motivation, challenges, and practical solutions for building a real‑time data warehouse using Flink and Iceberg.

Background and Pain Points

High‑volume applications (e.g., mini‑programs and video accounts) generate petabyte‑scale data daily, leading to complex data platform architectures with many components (Kafka/RocketMQ, Flume/NiFi/DataX, Hive/HBase/HDFS/S3, Spark, MapReduce, Flink, MySQL, Elasticsearch, Presto, etc.). The traditional Lambda architecture separates offline batch and online stream layers, causing duplicated pipelines, high operational cost, data inconsistency, and long latency.

Kappa Architecture and Its Issues

Kappa replaces the batch layer with a unified stream built on a message queue, reducing duplication but introducing new problems: reliance on queue ordering, lack of OLAP optimizations, and difficulty handling updates.

Iceberg Overview

Iceberg is an open table format that sits between compute engines (Flink, Spark) and storage (Parquet, Avro, ORC) and provides snapshot‑based read/write isolation, stream‑batch unified access, ACID semantics, schema evolution, and partition handling.

Iceberg Capabilities

Snapshot‑based read/write separation and time‑travel.

Support for Parquet, Avro, ORC file formats.

ACID guarantees and multi‑version data.

Table, schema, and partition evolution.

Read/Write Process

Writes create a tentative snapshot that becomes visible only after a commit; multiple snapshots can be read concurrently, enabling incremental reads by scanning only the new data between snapshots.

Flink‑Iceberg Integration

The FlinkIcebergSink writes data files (DataFiles) via IcebergStreamWriter, then a single IcebergFileCommitter atomically commits them, turning invisible files into visible snapshots. To avoid the small‑file problem, a data‑compaction operator merges tiny files periodically, and the commit operation remains lightweight.

Real‑Time Warehouse Construction

By leveraging Iceberg’s snapshot isolation and Flink’s streaming capabilities, near‑real‑time ingestion (seconds‑to‑minutes latency) is achieved, while still supporting batch‑style OLAP queries via Presto or Spark. Iceberg can replace Kafka as the central storage layer, offering unified stream‑batch processing, efficient back‑tracking, and lower storage costs.

Advantages and Trade‑offs

Unified storage for stream and batch.

OLAP‑friendly columnar formats enable predicate push‑down.

Efficient time‑travel for back‑tracking.

Reduced storage cost compared to pure message‑queue solutions.

Latency is near‑real‑time rather than true real‑time.

Additional integration effort required for other systems.

Best Practices

Enable automatic small‑file compaction via Iceberg table properties, use SQL extensions to manage snapshots, delete obsolete files, and perform periodic compaction. Configure Flink to read incrementally from a specific snapshot, and use SQL‑based pipelines for end‑to‑end data processing.

Future Plans

Enhance Iceberg core with row‑level deletes and richer SQL extensions.

Build a unified index (e.g., Bloom filter) for faster file lookup.

Integrate Alluxio as a caching layer for sub‑second analytics.

Automate schema discovery, metadata management, and cross‑system integration.

The presentation concludes with a thank‑you and invites readers to follow, like, and share the content.