Implementing an Integrated Data Lake and Lakehouse Architecture with Apache Iceberg and Flink

What Is a Data Lake?

A data lake is a centralized repository that stores massive amounts of raw‑format data using a flat, object‑storage architecture. Unlike hierarchical data warehouses, a lake stores data with metadata tags and unique identifiers, enabling cheap, scalable storage of structured, semi‑structured, and unstructured data (images, video, audio, documents). This openness makes it suitable for modern machine‑learning and advanced‑analytics workloads.

Why Use a Data Lake?

Open format : avoids vendor lock‑in of proprietary warehouses.

Low cost and high scalability : leverages inexpensive object storage.

Supports all data types : essential for ML and analytics.

Self‑service : analysts, data scientists, and engineers can query raw data without a predefined schema.

Challenges of Data Lakes

Despite the benefits, data lakes suffer from reliability, performance, and security issues. Without proper tooling, data quality can degrade, query performance slows as data volume grows, and lack of governance makes compliance difficult.

Specific challenges

Reliability problems such as corrupted batches and difficulty merging streaming and batch data.

Performance degradation caused by metadata management and poor partitioning.

Missing security and governance features, preventing fine‑grained access control.

Lakehouse as a Solution

A lakehouse adds a transactional storage layer on top of the data lake, combining the flexible storage of a lake with the management capabilities of a warehouse. It enables SQL analytics, data‑science workloads, and BI reporting on the same open‑format data without moving it.

Case Study: AutoDTS Lakehouse Implementation

The following practice is based on a real‑world project (AutoDTS) that builds a lakehouse on Apache Iceberg.

1. Iceberg Key Features

ACID semantics.

Incremental snapshot mechanism.

Open table format.

Streaming‑batch interface support.

2. Data Ingestion Pipelines

Append stream into the lake : Log data from client, user, and server logs are sent to Kafka, then Flink writes them into Iceberg tables stored on HDFS.

Flink‑SQL integration (Flink 1.11 + Iceberg 0.11):

Meta Server extended to support Iceberg catalog.

SQL SDK extended for Iceberg catalog.

Platform exposes Iceberg table management, allowing users to create tables via SQL.

3. Proxy‑User Support

To respect budget and permission boundaries, the platform adds a table‑level property iceberg.user.proxy='targetUser' and enables:

Super‑user mode.

Team‑account authentication.

HDFS access with proxy user.

Hive Metastore access with proxy user (referencing Spark’s HiveDelegationTokenProvider).

4. CDC (Change‑Data‑Capture) Ingestion

CDC streams from MySQL, SQLServer, TiDB are captured by AutoDTS, sent to Kafka, and then written to Iceberg using the same Flink pipeline.

5. Copy‑On‑Write Sink

Flink 1.11 originally provided an AppendStreamTableSink that could not handle CDC. A custom Copy‑On‑Write sink was implemented, featuring:

Multi‑parallel StreamWriter for data writes.

Single‑parallel FileCommitter for ordered commits.

Bucket configuration via partition.bucket.source='id' and partition.bucket.num='10' to guarantee that the same record lands in the same bucket.

Support for primary keys (PR1978) and Iceberg format version 2.

6. Computing Engines

Flink : Core real‑time engine for data‑in‑lake scenarios; supports both streaming and batch queries.

Hive : Provides batch SQL integration with Iceberg and Spark 3, enabling table‑level metadata queries such as SELECT * FROM db.table.history to view snapshots, files, and manifests.

Trino/Presto : Integrated for BI reporting; requires metadata caching (see GitHub issue #7551) to achieve low‑latency queries on Iceberg tables.

7. Pitfalls Encountered

The project faced issues such as small‑file explosion, missing bucket support in Flink SQL, and the need to manually manage snapshot state during checkpoints. These were mitigated by the custom Copy‑On‑Write sink, bucket configuration, and careful checkpoint handling.

8. Benefits and Conclusions

Lakehouse integration : Iceberg tables behave like Hive tables, offering the same data format and compute engine while adding transactional guarantees.

Stream‑batch convergence : Same source, compute, and storage for near‑real‑time analytics (minute‑level data visibility).

Business impact : Unified data platform reduces data silos, shortens ETL cycles, and enables both batch and real‑time BI without data duplication.

Future roadmap : Continue to improve real‑time data warehouse capabilities, enhance metadata caching for Trino, and automate small‑file compaction.