Choosing the Right Open‑Source Data Lake: Delta vs Iceberg vs Hudi

Background

Data lakes built on classic Lambda architecture and plain Parquet files suffer from schema drift, lack of ACID guarantees, inefficient upserts, small‑file problems, and limited streaming support.

Delta Lake (Databricks)

Delta Lake adds a transaction log to Parquet, providing ACID transactions, schema enforcement, time‑travel, snapshot isolation, and efficient upserts/deletes. It is designed for Apache Spark and unifies batch and streaming workloads on a single storage layer.

Apache Hudi (Uber)

Hudi was created to support fast upserts, deletes and incremental consumption for Uber’s ride‑order pipeline. It offers two storage types—Copy‑On‑Write (CoW) and Merge‑On‑Read (MoR). CoW rewrites whole files on update; MoR writes delta files that are later compacted. Hudi provides three read views: base‑only, delta‑only, or merged, enabling both batch and streaming consumption.

Apache Iceberg (Netflix)

Iceberg was developed to overcome Hive’s partition explosion, metadata latency, and lack of atomicity. It defines a highly abstracted table format with an independent schema, supports multiple compute engines, and stores metadata in a separate manifest list. While its feature set is smaller than Delta or Hudi, the design enables engine‑agnostic data lake operations.

Common Requirements Addressed

All three projects aim to provide:

ACID guarantees and snapshot isolation

Schema evolution with validation

Efficient upserts and deletes

Streaming ingestion and incremental reads

File‑system independence

Optimized query performance

Seven‑Dimension Comparison

ACID & Isolation – Snapshot isolation offers the best concurrency; Delta and Hudi provide strong guarantees; Iceberg is adding comparable support.

Schema Evolution – Iceberg abstracts schema; Hudi supports additive/nullable changes; Delta enforces schema at write time.

Streaming Support – Delta and Hudi support streaming reads; Iceberg currently lacks native streaming (under development).

Abstraction & Pluggability – Iceberg is engine‑agnostic; Delta tightly couples to Spark; Hudi couples to Spark/Flink.

Query Performance – Delta benefits from Spark optimizations; Iceberg relies on external engines; all provide file‑level pruning and metadata caching.

Additional Features – Delta offers Python APIs and easy demos; Iceberg adds file‑level encryption; Hudi includes built‑in compaction and fast upserts.

Community Activity (early 2020) – Delta and Hudi have vibrant open‑source communities and commercial backing; Iceberg activity is primarily on GitHub issues and pull requests.

Key Design Details

Delta Lake Transaction Log

Each commit writes a JSON file to _delta_log/ containing added and removed files. Readers reconstruct the latest snapshot by scanning the log, enabling time‑travel queries via VERSION AS OF or TIMESTAMP AS OF.

Hudi Write Paths

CoW writes a new Parquet file for each update; MoR writes delta log files (e.g., .log) that are later compacted into base files. Compaction can be scheduled or triggered manually.

Iceberg Metadata

Iceberg stores table metadata in a manifest list; each manifest references data files with statistics (min/max values, row count). This enables predicate push‑down to the file level without an external metastore.

Conclusion

Delta Lake provides a Spark‑centric, feature‑rich table format with strong ACID guarantees. Iceberg offers a modular, engine‑agnostic foundation suitable for multi‑engine environments, though upsert support is still evolving. Hudi focuses on fast upserts and incremental consumption, making it a good fit for pipelines that require frequent data corrections and streaming reads.