What Kind of Data Lake Do Enterprises Really Need? Lessons from Delta 2.0

Industry Highlights and Delta 2.0 Release

Delta 2.0’s open‑source launch sparked discussion after Databricks presented a performance comparison that positioned Delta against Iceberg and Hudi. The announcement highlighted a key feature: conversion from Iceberg to Delta, with Adobe’s migration case study emphasized.

Table‑Format Competition

Table formats define the files that constitute a table, enabling any engine to query data and providing ACID guarantees, schema evolution, and other advanced functions. The three dominant open‑source formats are Delta, Iceberg, and Hudi.

Delta

Initiated by Databricks in 2017, open‑sourced in 2019, Delta was created to address transactional, streaming, and BI shortcomings of traditional data lakes. Databricks promoted the “lakehouse” concept, which combines data‑lake scalability with data‑warehouse reliability. Gartner’s 2021 report placed Databricks and Snowflake in the leading quadrant, noting lakehouse technology is still 3‑5 years from full maturity.

Delta 1.0 aimed to replace the Lambda architecture by allowing batch and streaming workloads on Spark, but Spark’s limited adoption in China and tight coupling to Spark constrained community growth, giving Iceberg an early advantage.

Iceberg

Developed by Netflix and graduated to Apache in 2020, Iceberg offers ACID and MVCC, data‑skipping, efficient planning, S3‑first design, schema evolution, and hidden partitions. It is widely supported by Cloudera, Snowflake, StarRocks, and Amazon Athena, and was the first format to provide a Flink connector.

The author first encountered Iceberg in 2020 while seeking a better lake solution for Flink, noting its cautious roadmap and engine‑agnostic design.

Hudi

Hudi (Hadoop Upsert and Incremental) originated as a Spark‑centric upsert solution and has evolved into a broader platform with streaming upserts, CDC, and a self‑managing database layer. Its development pace and community direction differ markedly from Iceberg’s steady approach.

Benchmark Findings

Databeans’ third‑party test reported Delta 2.0 is 1.7× faster than Iceberg and 4.3× faster than Hudi.

In‑house Trino TPCH tests on 100 warehouses showed Delta’s average response time about 1.4× faster than Iceberg under default settings.

Two configuration differences explained the gap: Delta uses SNAPPY compression (vs. Iceberg’s ZSTD) and a smaller default read‑target‑size (32 MiB vs. 128 MiB), which yields higher concurrency.

When both formats were aligned to SNAPPY compression and a 32 MiB read‑target‑size, response times converged, indicating the performance advantage stems mainly from configuration rather than inherent I/O speed.

Enterprise Data‑Lake Requirements

The author distills essential capabilities: structural freedom (schema changes without rewrites), read/write freedom (ACID guarantees), unified batch‑and‑stream processing, and engine‑agnostic support (Flink, Spark, Trino, etc.). Real‑world practice shows challenges such as CDC replacing message queues (introducing small‑file issues) and the need for read‑time merging to compete with dedicated real‑time warehouses.

Enterprises must avoid fragmented pipelines that require separate batch and streaming tables, instead adopting a lakehouse that unifies ETL, data pipelines, and OLAP.

Arctic Project

To address these needs, the team open‑sourced Arctic, a streaming lake‑house service built on Iceberg. Arctic offers:

Self‑optimizing capabilities.

Dual compatibility mode (Hive or Iceberg) for seamless migration.

Concurrent multi‑engine writes with primary‑key conflict resolution.

Standardized metrics, management tools, and a Thrift API.

Benchmarking Arctic against Hudi in an HTAP‑style workload (Flink CDC writes, TPCH read‑time merging) showed Arctic achieving lower latency (smaller numbers indicate better performance). Detailed test methodology and results are published on the Arctic website.

Conclusion

Delta 2.0’s open‑source release clarifies the emerging standard for data‑lake table formats, intensifying competition among Delta, Iceberg, and Hudi. Enterprises should evaluate these formats based on the four core capabilities and consider open‑source projects like Arctic that extend lakehouse functionality to real‑time streaming while preserving compatibility and self‑optimization.