Tagged articles
Delta Lake
59 articles · Page 1 of 1
This article provides a comprehensive, step‑by‑step guide to mastering Spark SQL in high‑concurrency, billion‑row scenarios, covering the execution chain, architecture layers, Delta Lake integration, performance tuning, production‑grade streaming and batch pipelines, Kubernetes deployment, parameter management, observability, and real‑world case studies.
Data EngineeringDelta LakeKubernetes
0 likes · 44 min read

This roundup compiles the latest releases across major data platforms—including ClickHouse, Apache Fluss, Delta Lake, StarRocks, Apache Pulsar and DolphinScheduler—highlighting version numbers, key feature additions, security fixes, and emerging trends shaping the big‑data ecosystem.
Apache FlussClickHouseData Engineering
0 likes · 19 min read

Apache XTable introduces a lightweight metadata translation layer that decouples data storage from format metadata, enabling zero‑copy, omni‑directional conversion among Hudi, Iceberg, and Delta Lake, allowing organizations to write with one format and read with any engine without duplicating Parquet files.
Apache XTableDelta LakeHudi
0 likes · 7 min read

This article provides a detailed technical comparison of Apache Hudi, Delta Lake, and Apache Iceberg, covering core features such as incremental pipelines, concurrency control, merge‑on‑read, partition evolution, multi‑mode indexing, ingestion tools, and real‑world use cases, and explains why Hudi often leads for heavy update workloads.
Apache HudiApache IcebergDelta Lake
0 likes · 14 min read

Amid growing data demands, this article explains the data lake technology maturity curve, detailing lake‑warehouse architectural patterns, design principles, core functionalities, and the four leading open‑source solutions (Hudi, Iceberg, Delta Lake, Paimon) to guide enterprises in building flexible, scalable, and governed data platforms.
Delta LakeHudiIceberg
0 likes · 10 min read

This article explains the data lake technology maturity curve, covering lake‑warehouse architecture patterns, design principles, core capabilities of major open‑source lake engines (Hudi, Iceberg, Delta Lake, Paimon), and practical application scenarios for modern data‑driven enterprises.
Delta LakeHudiIceberg
0 likes · 10 min read

This article explains the rapid growth of data-driven businesses, the challenges of traditional data warehouses, and how modern data lake technologies such as Delta Lake, Hudi, Iceberg, and Paimon form a maturity curve that guides enterprises in architecture choices, design principles, core capabilities, and practical applications.
Delta LakeHudiIceberg
0 likes · 12 min read

Databricks announced the acquisition of Tabular, the company founded by the original creators of Apache Iceberg, aiming to integrate Delta Lake and Iceberg into a unified, open lakehouse architecture that enhances format compatibility, reduces data silos, and supports AI workloads.
Apache IcebergDatabricksDelta Lake
0 likes · 5 min read

Databricks announced its acquisition of Tabular, the creators of Apache Iceberg, aiming to unify lakehouse formats through Delta Lake UniForm, while highlighting the rise of lakehouse architecture, format fragmentation, and the push toward open data interoperability.
Apache IcebergDatabricksDelta Lake
0 likes · 9 min read

This article introduces Delta Lake 3.1, detailing its release background, the addition of Deletion Vector to Update and Merge commands, metadata‑driven count/min/max optimizations, the Universal Format for cross‑engine compatibility, and a comparative evaluation with Iceberg and Hudi.
Deletion VectorDelta LakeUniversal Format
0 likes · 8 min read

This article examines the design of lakehouse storage systems by comparing Delta Lake, Apache Hudi, and Apache Iceberg, focusing on metadata management, Merge‑On‑Read mechanisms, and a series of query and write performance optimizations with real‑world EMR case studies.
Apache HudiApache IcebergDelta Lake
0 likes · 16 min read

This article compares traditional data warehouses, modern data lakes, and emerging lakehouse architectures, explaining their design patterns, advantages, disadvantages, and suitable use cases, while detailing implementation considerations such as schema design, ETL/ELT processes, file formats like Delta, Iceberg, and Hudi, and factors influencing platform selection.
Apache SparkDelta LakeETL
0 likes · 20 min read

This article analyzes the rise of lake‑house architecture in the Hadoop ecosystem, compares the technical capabilities of Hudi, Iceberg and Delta Lake, details implementation enhancements such as MOR and multi‑writer support, showcases Flink integration, presents a real‑time marketing use case, and outlines future development directions.
Delta LakeHadoopHudi
0 likes · 14 min read

Guandata’s R&D leader outlines how their analytics platform leverages Delta Lake and Spark to deliver fast, ACID‑compliant BI and AI workloads, detailing architecture, key features like schema evolution and time travel, and practical performance tricks such as compaction, vacuuming, and multi‑engine integration.
AIBIDelta Lake
0 likes · 14 min read

This article examines the architecture and core design of lakehouse storage systems, compares the metadata handling and Merge‑On‑Read mechanisms of Delta Lake, Apache Hudi, and Apache Iceberg, and presents practical performance‑optimization techniques and real‑world case studies on Alibaba Cloud EMR.
Apache HudiApache IcebergDelta Lake
0 likes · 18 min read

This article summarizes a technical presentation on how Delta Lake is integrated into a BI+AI platform, covering the product background, data‑lake architecture, Delta Lake features such as ACID transactions, schema management, multi‑engine support, performance optimizations, and future development directions.
AIBIDelta Lake
0 likes · 12 min read

This interview with NetEase data‑lake technology manager Ma Jin explains the distinction between data lakes and lakehouses, reviews the evolution of table‑format technologies such as Iceberg, Hudi and Delta Lake, evaluates feature maturity and performance trade‑offs, and discusses systematic versus non‑systematic adoption in enterprises.
Delta LakeHudiIceberg
0 likes · 13 min read

This article explains how Inceptor and Apache Delta Lake combine distributed transaction, MVCC, snapshot isolation, and high‑performance SQL to support both data lake and data warehouse workloads, compares them with Hudi and Iceberg, and outlines their strengths and limitations for modern big‑data analytics.
Delta LakeInceptordata lake
0 likes · 12 min read

This interview with NetEase’s data‑lake technology manager explores the distinction between data lakes and lakehouses, the evolution of table‑format technologies such as Iceberg, Hudi and Delta Lake, their maturity across key capabilities, and the practical adoption challenges faced by enterprises.
Delta LakeHudiIceberg
0 likes · 14 min read

This article reviews lake‑format concepts, Apache Hudi architecture, CDC fundamentals, design considerations for CDC on lake formats, implementation details of Hudi CDC, and streaming optimizations including automated lake‑table management and a simplified StreamingSQL for Spark.
Apache HudiCDCDelta Lake
0 likes · 19 min read

This article explains the Lakehouse concept, why it is needed, the limitations of traditional data warehouses and data lakes, and how Databricks’ unified architecture—through open storage formats, fine‑grained governance, and optimized query engines—delivers high‑quality, low‑latency data for BI, analytics, and machine learning workloads.
Data EngineeringDatabricksDelta Lake
0 likes · 21 min read

This article examines the technical differences and feature sets of Apache Hudi, Delta Lake, and Apache Iceberg, highlighting incremental pipelines, concurrency control, merge‑on‑read storage, partition evolution, multi‑mode indexing, and real‑world use cases to help practitioners choose the most suitable lakehouse solution for their workloads.
Apache HudiApache IcebergDelta Lake
0 likes · 18 min read

The article examines the open‑source release of Delta 2.0, compares its features and benchmark results with Iceberg and Hudi, discusses the core capabilities required by enterprises for a lakehouse architecture, and introduces the Arctic project as a multi‑engine streaming lake service.
ArcticDelta LakeHudi
0 likes · 25 min read

The article reviews recent developments in data‑lake table formats—Delta Lake 2.0, Iceberg, and Hudi—examining their features, benchmark results, and ecosystem impact, and then introduces Arctic, an open‑source streaming lakehouse service built on Iceberg that aims to bridge batch‑stream gaps for enterprises.
Delta LakeHudiIceberg
0 likes · 24 min read

This talk by eBay software engineer Zhu Feng explains the fundamentals of Delta Lake and Lakehouse architecture, outlines eBay’s migration from Teradata to a Spark‑based platform, and details the custom enhancements, performance optimizations, and operational improvements implemented to support large‑scale update and delete workloads.
Delta Lakedata lakeeBay
0 likes · 16 min read

This article introduces Delta Lake as an open‑source lakehouse storage framework, explains its core features, file and metadata structures, details Alibaba Cloud EMR's enhancements and deep integration with DLF, and presents G‑SCD and CDC solutions for real‑time incremental data warehousing.
CDCDLFDelta Lake
0 likes · 11 min read

This article introduces Delta Lake as an open‑source storage layer for lake‑house architectures, explains its key features, file and metadata structures, and details how Alibaba Cloud EMR and Data Lake Formation integrate and extend Delta Lake with advanced capabilities such as G‑SCD, CDC, performance optimizations, and future roadmap.
CDCDLFDelta Lake
0 likes · 10 min read

This article explains how Delta Lake adds reliability to data lakes by offering ACID transactions, scalable metadata, and unified batch‑and‑stream processing, outlines the challenges it solves, details its implementation principles, and demonstrates a practical demo for building an integrated data warehouse.
ACIDData EngineeringDelta Lake
0 likes · 9 min read

Delta Lake, an open‑source storage layer from Databricks, provides ACID transactions, data versioning, schema evolution, and unified batch‑stream processing, with a detailed file structure and metadata mechanism, while Alibaba Cloud EMR enhances it with advanced DML, performance optimizations, deep DLF integration, and solutions for G‑SCD and CDC.
CDCDLFDelta Lake
0 likes · 11 min read


ITPUBApr 26, 2022 · Big Data
This article explains the fundamentals of data lakes and data warehouses, compares their architectures, outlines the challenges of data lakes, and then dives deep into Delta Lake's core features, storage model, ACID guarantees, concurrency handling, and provides step‑by‑step Spark code examples for practical use.
ACIDDelta LakeMerge-On-Read
0 likes · 18 min read

This article explains the evolution from traditional data warehouses to data lakes and the modern Lakehouse architecture, introduces Delta Lake's core concepts, multi‑hop medallion tables, ACID transactions, generated columns, standalone support, and future open‑source directions.
Data ManagementDelta LakeGenerated Columns
0 likes · 13 min read

This article details how ZuoYeBang's engineering team migrated their Hive‑based offline data warehouse to Delta Lake, tackling latency, scalability, and query‑performance challenges through stream‑to‑batch processing, data‑lake architecture, and optimizations like DPP and Z‑ordering.
Delta LakeHivePerformance Optimization
0 likes · 15 min read

NetEase Yanxuan replaced its cumbersome data‑warehouse with a flexible Delta‑Lake/Iceberg data lake, creating a unified metadata layer and real‑time ingestion pipelines that cut latency from nightly batches to seconds, slashed compute and storage costs, supported diverse business scenarios and machine‑learning feature engineering, and set the stage for broader future expansion.
Delta LakeFlinkIceberg
0 likes · 16 min read

This article explains how NetEase Yanxuan evolved from a traditional data‑warehouse pipeline to a cloud‑native data‑lake architecture, detailing the business challenges, design choices, technology stack (Delta, Iceberg, Hudi), implementation steps, and the resulting gains in real‑time data access, cost reduction, and feature‑engineering support.
Delta LakeHudiIceberg
0 likes · 18 min read

This article provides a comprehensive overview of the Lakehouse paradigm, tracing its origins from traditional data warehouses and data lakes, comparing architectures, detailing core components such as Delta Lake and Iceberg, and illustrating practical cloud implementations and future directions.
Apache IcebergCloud Data PlatformData Management
0 likes · 14 min read

This article details Baicaowei's migration from an IDC‑hosted Hadoop cluster to a cloud‑native data lake on Alibaba Cloud, outlining the business drivers, pain points of the legacy platform, architectural goals, design principles, solution selection, implementation steps, and future outlook for the new big‑data ecosystem.
Alibaba CloudCloud MigrationDelta Lake
0 likes · 16 min read

This article provides a comprehensive overview of Snowflake and Delta Lake, detailing their cloud‑native architectures, storage‑compute separation, transaction support, performance optimizations, and a side‑by‑side comparison of their capabilities, openness, and real‑time data handling.
ACIDCloud ArchitectureDelta Lake
0 likes · 25 min read

This article provides an in-depth overview of data lake concepts, definitions, and essential features, followed by detailed case studies of enterprise data lake implementations and comparative analysis of leading data lake table formats—Iceberg, Hudi, and Delta Lake—highlighting their architectures, capabilities, and trade‑offs.
Delta LakeFlinkHudi
0 likes · 19 min read

This article examines JD's shift from a traditional Lambda‑based data warehouse to a Delta Lake‑powered real‑time data lake, detailing the challenges of legacy architectures, the evaluation of open‑source table formats, Delta Lake's core mechanisms, and the resulting simplified batch‑stream development workflow.
Batch-StreamDelta Lakebig data
0 likes · 11 min read

This article explains the rapid growth of data, the limitations of traditional warehouses, and how a cloud‑based data lake built on object storage with Delta Lake format provides low‑cost, flexible, and ACID‑compliant analytics, followed by a step‑by‑step guide to ingest, manage, and analyze data using Alibaba Cloud DLF and Databricks DDI with Spark streaming and batch jobs.
Alibaba CloudDelta LakeSpark
0 likes · 19 min read

This article compares Apache Hudi, Apache Iceberg, and Delta Lake, examining their storage formats, platform compatibility, update performance, concurrency guarantees, and integration with lakeFS to help readers choose the most suitable solution for their data lake use case.
Apache HudiApache IcebergConcurrency
0 likes · 16 min read

This article describes how Soul's data engineering team replaced nightly batch ETL with real-time Delta Lake ingestion on EMR, detailing the motivations, comparative analysis of Delta, Hudi, Iceberg, the implementation architecture, encountered issues such as data skew and schema evolution, and the solutions adopted to improve performance and reliability.
Data SkewDelta LakeEMR
0 likes · 13 min read

This article presents a comprehensive overview of data lake implementations, detailing Huawei's production‑scene platform, a real‑time financial data lake architecture using Kafka, Flink and Iceberg, and Soul's Delta Lake practice with Spark, Hive, and custom ETL tools, highlighting design choices, processing flows, and operational considerations.
Delta LakeFlinkHive
0 likes · 8 min read

This article explains what a data lake is, outlines its key characteristics, and compares three major data lake frameworks—Iceberg, Hudi, and Delta Lake—highlighting their architectures, features, and trade‑offs for large‑scale data storage and processing.
Delta LakeHudiIceberg
0 likes · 13 min read

This article describes how a data engineering team replaced nightly batch ETL with a Delta Lake‑based real‑time log ingestion pipeline on EMR, detailing the motivations, architecture, implementation steps, encountered issues such as data skew and schema evolution, and the practical solutions they applied to achieve low‑latency, reliable data delivery.
Delta LakeHiveSpark
0 likes · 14 min read

This article describes how FluentSpeak replaced a DataX master‑slave pipeline with a CDC‑plus‑Delta Lake solution on Alibaba Cloud EMR, detailing architecture choices, streaming SQL merge logic, monitoring, challenges, and the resulting cost and latency improvements.
CDCDelta LakeEMR
0 likes · 17 min read

This article introduces JD's real‑time data warehouse evolution, outlines the limitations of traditional Lambda‑based warehouses, compares open‑source lake formats (Delta, Hudi, Iceberg), explains Delta Lake's transaction‑log architecture and read flow, and demonstrates how a unified batch‑stream development model simplifies data processing and improves reliability.
ACIDDelta Lakedata lake
0 likes · 12 min read

The article describes Alibaba Cloud's Data Lake Formation service, presenting a unified, real‑time, and low‑latency solution for ingesting heterogeneous data sources—including RDS, DTS, TableStore, and SLS—into an OSS‑backed data lake using templates, a Spark‑based ingestion engine, and modern file formats such as Delta Lake.
Alibaba CloudData IngestionDelta Lake
0 likes · 10 min read

This article examines the core requirements of data lakes and provides an in‑depth comparison of three major open‑source solutions—Apache Hudi, Apache CarbonData, and Delta Lake—highlighting their architectures, ACID support, query capabilities, and suitability for various real‑time and batch use cases.
ACIDApache CarbonDataApache Hudi
0 likes · 9 min read

Delta Lake adds ACID transaction support, schema enforcement, data versioning, and unified batch‑and‑stream processing to Apache Spark‑based data lakes, addressing reliability, quality, performance, and update challenges of traditional data lake architectures.
ACID transactionsApache SparkDelta Lake
0 likes · 13 min read

The article reviews the evolution of the big‑data ecosystem from 2017 to 2019, highlights Spark’s dominance, examines storage‑layer challenges of traditional Hive‑based warehouses, and explains how Delta Lake’s metadata‑driven library simplifies architecture, adds ACID features, and competes with Hudi and Iceberg.
Delta LakeSpark
0 likes · 8 min read

This article examines the three leading open‑source data‑lake projects—Delta Lake, Apache Iceberg, and Apache Hudi—by outlining their origins, core problems they address, key features, and a detailed seven‑dimension comparison to help practitioners choose the most suitable solution for their scenarios.
Apache HudiApache IcebergDelta Lake
0 likes · 17 min read

An in‑depth comparison of the three leading open‑source data lake platforms—Delta Lake, Apache Iceberg, and Apache Hudi—examines their origins, core challenges they address, key features, and performance across seven evaluation dimensions to guide practitioners in selecting the optimal solution for their workloads.
Apache HudiApache IcebergDelta Lake
0 likes · 15 min read

This article compares three open‑source data‑lake storage layers—Hudi, Iceberg, and Delta Lake—examining their shared reliance on meta‑files for schema and transaction handling, and detailing their differing designs for upserts, streaming support, query performance, and ecosystem integration.
Delta LakeHudiIceberg
0 likes · 13 min read

This article compares the design goals, data‑lake table formats—Hudi, Iceberg, and Delta—highlighting their common reliance on meta files and their distinct strengths for upserts, analytics, and unified streaming‑batch processing in modern big‑data environments.
Delta LakeHudiIceberg
0 likes · 10 min read

This article examines the root causes and performance impact of massive small-file proliferation in traditional data warehouses, explains why HDFS metadata limits scalability, and details how Delta Lake’s custom compaction process can safely merge these files for append-only tables without disrupting reads or writes.
Delta LakeHDFScompaction
0 likes · 5 min read

This article explains the origins and motivations of Delta Lake, details its ACID transaction support, schema enforcement, metadata handling, versioning, and unified batch‑and‑stream processing, and provides a step‑by‑step Maven and Spark code tutorial for creating, updating, and querying Delta tables.
ACIDApache SparkDelta Lake
0 likes · 10 min read

The article provides a comprehensive overview of recent Apache Spark advancements—including Delta Lake, Data Source V2, runtime optimizations, relational cache, cloud‑native challenges, AI integration via Project Hydrogen, and the anticipated features of Spark 3.0—highlighting how these innovations address modern data‑warehouse, cloud, and machine‑learning workloads.
Apache SparkDelta LakeRuntime Optimization
0 likes · 17 min read

The 2018 Spark+AI Summit in San Francisco showcased Spark's evolution toward unified AI and big‑data processing, introducing the Hydrogen project with gang scheduling, the open‑source MLflow platform, the Delta unified analytics engine, Spark 2.3 enhancements, and Facebook's shuffle I/O optimizations.
AIDelta LakeHydrogen
0 likes · 8 min read
