Evolution and Comparison of High‑Performance Cloud‑Native Lakehouse Storage Architecture: From HDFS to JuiceFS

Introduction This article explores high‑performance cloud‑native lakehouse storage architectures, covering their evolution, a comparison of HDFS and object storage, future storage options, and a practical implementation using JuiceFS.

1. Evolution of Storage Architecture The storage systems for big data have evolved from the on‑premise era, dominated by HDFS, to the cloud era where object storage (e.g., S3) is the primary choice for data lakes due to its scalability and support for diverse data types.

HDFS originated from Google File System in 2006, featuring a separate NameNode for metadata, multi‑replication, and compute‑storage coupling. It handles large files well but struggles with small files and scaling beyond billions of objects without architectural adjustments.

Object storage also appeared in 2006, designed for massive unstructured data. Its flat key‑value metadata, HTTP‑based API, low cost, and eventual consistency bring advantages and challenges, especially for metadata‑intensive big‑data workloads.

2. Comparison of Storage Systems The article compares HDFS and object storage across dimensions such as scale, metadata performance, operational complexity, and high‑availability. Object storage offers virtually unlimited scale but suffers from higher metadata latency and lower throughput for operations like directory rename.

HDFS’s single‑point NameNode is a bottleneck; extensions like Federation, Router‑based Federation, Standby NameNode, and JournalNode address scalability and HA.

Object storage metadata is flat KV; operations like renaming a large “directory” require recursive copy, index updates, and deletion, which are costly and lack strong consistency guarantees.

3. Future Lakehouse Storage Requirements A next‑generation lakehouse should provide extreme scalability, high availability, strong performance for both large and small files, cloud elasticity, compute‑storage separation, and efficient small‑file handling.

4. JuiceFS Overview JuiceFS is a strongly consistent distributed file system positioned as a drop‑in replacement for HDFS. Its architecture mirrors HDFS with separate metadata, data, and client layers, but stores data in object storage while using pluggable metadata engines (Redis, MySQL, TiKV, etc.). It decouples metadata from object‑storage metadata, supports horizontal scaling to billions of files, and includes caching for both metadata and data.

5. Practical Implementation on JuiceFS The lakehouse stack places JuiceFS (backed by object storage) as the unified storage layer, followed by data‑management layers such as Delta Lake, Hudi, or Iceberg, and then query engines, BI tools, and compute frameworks. Benchmarks show JuiceFS matches or exceeds HDFS performance for metadata latency, throughput, and TPC‑DS query workloads while outperforming raw object storage.

Q&A The article concludes with a Q&A covering topics such as the future of HDFS, limitations of S3, trade‑offs between compute‑storage separation and coupling, and differences between JuiceFS and Alluxio.