Exploring JuiceFS in Data Lake Storage Architecture

The talk begins with a review of the evolution of big‑data storage architectures, outlining three stages: early data warehouses, the emergence of data lakes to address data silos, and the recent lakehouse paradigm that combines the strengths of both.

It explains why data lakes are needed, emphasizing the challenges of data silos, diverse data formats, distributed data management, storage‑compute coupling, and AI workloads that require POSIX‑compatible access.

A definition of a data lake is provided, highlighting the importance of storing data in its natural/raw format, typically on cheap, scalable object storage, while keeping metadata in systems like Redis, MySQL, or TiKV.

The concept of lakehouse integration is introduced, describing how traditional data warehouses become post‑processing steps and how lakehouse architectures benefit from open file formats (Parquet, ORC) and open storage layers (Delta Lake, Iceberg, Hudi) combined with flexible compute engines.

JuiceFS is then introduced as an open‑source, cloud‑native distributed file system offering full POSIX, HDFS, and S3 API compatibility. Its architecture separates metadata (stored in databases) from data (persisted in object storage) and supports a wide range of underlying storage backends.

The presentation details JuiceFS’s internal design, including its chunk‑slice‑block hierarchy, configurable block size, and handling of small files without padding, as well as its caching, encryption, compression, and quota features.

Comparisons with HDFS and object storage show that JuiceFS matches HDFS performance for metadata operations while providing the elasticity and cost advantages of object storage, and it offers additional features such as atomic renames, concurrent writes, and strong consistency.

Further analysis of lakehouse requirements demonstrates how JuiceFS satisfies critical file‑system properties like atomic visibility, mutual exclusion, and consistent listing, and how its multi‑prefix block layout mitigates object‑storage API request limits and costs.

Benchmark results illustrate that, with cache warm‑up, JuiceFS delivers performance comparable to HDFS for TPC‑DS workloads using ORC and Parquet formats.

The talk concludes with JuiceFS’s ecosystem integrations, including native support for Hudi, Iceberg, Delta Lake, and the Fluid project for AI model training, as well as community initiatives such as the JuiceFS community story collection and upcoming events.