Comparing Apache Hudi, Apache Iceberg, and Delta Lake for Data Lake Storage

Introduction

When building a data lake, the choice of storage format is perhaps the most significant decision, directly affecting performance, usability, and compatibility.

By simply changing the storage format, we can unlock new capabilities and improve system performance.

Apache Hudi, Apache Iceberg, and Delta Lake are currently the best formats designed for data lakes, each addressing the most pressing challenges.

Atomic transactions – guarantee that updates or appends to the lake do not fail midway, avoiding dirty data.

Consistent updates – prevent read failures or incomplete results during writes and handle concurrent write conflicts.

Scalable data and metadata – avoid bottlenecks in object‑store APIs and metadata when tables grow to thousands of partitions and billions of files.

We will analyze each format’s approach to update performance, concurrency, and tool compatibility, and finally recommend the most suitable format for your data lake.

Platform Compatibility

Hudi was open‑sourced by Uber and is designed for incremental updates using columnar formats. It ingests data from multiple sources, primarily Apache Spark and Apache Flink, and provides a Spark‑based utility (DeltaStreamer) to read external resources such as Apache Kafka.

It supports reads from Apache Hive, Apache Impala, and PrestoDB, and includes a HiveSyncTool to sync Hudi table schemas to the Hive Metastore. (The Hudi community also supports DLA.)

Iceberg

Iceberg was originally released by Netflix to solve performance, scalability, and manageability challenges when storing large Hive‑partitioned datasets on S3.

Iceberg supports read and write in Apache Spark, including Structured Streaming, and is readable by Trino (PrestoSQL) with limited delete support. It also supports read/write in Apache Flink and provides read support for Apache Hive.

Delta Lake

Delta Lake is maintained as an open‑source project by Databricks, the creators of Apache Spark, and therefore has deep integration with Spark for both reads and writes.

Using Hive’s SymlinkTextInputFormat, it provides read support for Presto, AWS Athena, AWS Redshift Spectrum, and Snowflake, though each Delta table partition must export a symlink.txt file, which can become costly for large tables.

Update Performance & Throughput

Supporting row‑level updates on large immutable objects can be achieved in several ways, each with its own performance and throughput trade‑offs.

We will examine each format’s strategy for UPSERT operations and discuss additional optimizations related to read performance.

Hudi

Hudi’s UPSERT performance trade‑offs are explicit and differ between the two table types.

Copy‑on‑Write tables write updates fully into columnar files, creating new objects. This raises write cost but eliminates read amplification, making it ideal for read‑heavy workloads. Merge‑on‑Read tables write updates immediately to row‑based log files and periodically merge them into columnar Parquet files; queries can include the latest log data (snapshot view) or exclude it (read‑optimized view), offering flexibility between latency and query efficiency.

Hudi further optimizes compression by leveraging a key index to efficiently track files containing stale records.

Iceberg

Since the release of Spark 3.0, Iceberg supports UPSERTs via MERGE INTO queries, which rewrite files containing records to be updated.

Iceberg’s strength lies in read performance for tables with many partitions. By maintaining manifest files that map objects to partitions and storing column‑level statistics, Iceberg avoids expensive object‑store directory listings or Hive partition enumeration.

Additionally, Iceberg’s manifests allow a file to belong to multiple partitions, enabling efficient partition pruning and improving latency for highly selective queries.

Delta Lake

During a MERGE operation, Delta uses metadata‑driven data skipping to classify files for insert, update, or delete, then records these actions in a JSON “Delta Log”. Every ten commits, the log is rewritten as a Parquet checkpoint file that captures the table’s state, preventing costly log traversal.

To maintain performance, Delta tables require periodic compaction that merges many small Parquet files into larger ones (ideally ~1 GB, at least 128 MB). Databricks’ proprietary Delta Engine automates this compaction and provides additional write optimizations.

Delta Engine further improves performance with Bloom filters for key indexing, Z‑ordering for better file pruning at read time, and local caching.

Concurrency Guarantees

In‑place updates require handling concurrency.

If a table is read while being updated, or multiple writers attempt conflicting modifications, the system must resolve conflicts.

Traditional databases often use Multi‑Version Concurrency Control (MVCC) with an append‑only logical transaction log.

Optimistic Concurrency Control (OCC) allows concurrent writes and checks for conflicts at commit time, retrying failed transactions.

Hudi

Hudi provides both MVCC and OCC.

Its MVCC requires all writes to be totally ordered in a central log, limiting write concurrency to one writer at a time.

To overcome this, Hudi offers OCC, which relies on Apache Zookeeper or Hive Metastore to lock individual files and provide isolation.

Iceberg

Iceberg supports OCC by atomically swapping metadata files during updates.

Each write creates a new table “snapshot”. The writer then attempts a compare‑and‑swap (CAS) on a special value holding the current snapshot ID; if no other writer has replaced the snapshot, the operation succeeds, otherwise it retries.

On distributed file systems like HDFS this can be done locally; on S3 an additional component (currently only Hive Metastore) is required to store pointers.

Delta Lake

Delta uses Optimistic Control, as most lake operations append to time‑ordered partitions, avoiding conflicts.

When two processes add commits to the Delta Log, Delta silently checks for overlapping file changes and can allow both to succeed.

This requires the underlying object store to provide a CAS operation or a way to fail writes when concurrent updates occur.

Like Iceberg, this works out‑of‑the‑box on HDFS but not on S3, so Delta on AWS does not support multi‑cluster writes and lacks true transactional guarantees, unless the proprietary Delta Engine with Databricks‑managed external sync servers is used.

So which one is right for you?

We have examined the key similarities and differences among Apache Hudi, Delta Lake, and Apache Iceberg.

Now decide which format best fits your use case.

Go with Iceberg if…

Your primary pain point is managing metadata for massive tables (over 10 k partitions) on object storage; Iceberg alleviates performance issues related to S3 object listing or Hive Metastore partition enumeration.

Conversely, delete and change support is still early and incurs data‑retention overhead.

Go with Hudi if…

You use multiple query engines and need flexible management of changing datasets. Note that tool support and overall developer experience may be poorer, and large‑scale production deployments can require operational overhead.

If you are on AWS managed services such as Athena, Glue, or EMR, Hudi is pre‑installed, configured, and supported by AWS.

Go with Delta Lake if…

You are primarily a Spark shop and expect relatively low write throughput. If you are already a Databricks customer, Delta Engine provides significant read/write performance and concurrency improvements, making dual development within their ecosystem worthwhile.

For other Apache Spark distributions, Delta Lake remains open‑source but may lag behind Delta Engine as a product differentiator.

Integration With lakeFS

If you wonder “Can I use lakeFS with these formats?” the answer is yes.

lakeFS can coexist with any of Delta, Iceberg, or Hudi, providing branching, merging, and rollback capabilities across any number of tables.

Because these formats operate at the table level, they do not guarantee cross‑table consistency. lakeFS enables you to modify multiple tables on an isolated branch and then atomically merge those changes into a main branch, achieving cross‑table consistency.

Further Reading

ARTICLE: Diving Into Delta Lake: Unpacking the Transaction Log

DOCS: Branching Recommendations: Cross Collection Consistency

VIDEO: A Thorough Comparison of Delta Lake, Iceberg and Hudi

ARTICLE: In-depth Comparison of Delta, Iceberg, and Hudi

BLOG: Efficient Upserts into Data Lakes with Databricks Delta

ARTICLE: Comparison of Big Data storage layers: Delta vs Apache Hudi vs Apache Iceberg

This article was contributed to by Paul Singman, Developer Advocate at lakeFS.

Thank you for reading.