JuiceFS: A Cloud‑Native Distributed File System for Big Data and AI Workloads

Author: Su Rui, JuiceFS partner. Source: JuiceData.

In this talk, Su Rui introduces JuiceFS, an open‑source cloud‑native distributed file system released in January, which has attracted over 3,100 GitHub stars and appeared on Hacker News and GitHub Trending.

The presentation is divided into four parts: why a file system is needed in cloud‑native environments, the challenges of traditional object storage, JuiceFS’s design goals and architecture, and future plans.

First, the evolution of file storage over the past 40 years is reviewed, from proprietary hardware appliances to the rise of object storage (e.g., Amazon S3) and the limitations of object storage for big‑data analytics and AI workloads.

JuiceFS was started in 2017 to bring a POSIX‑compatible file system to cloud‑native settings, leveraging existing object storage for data and adding a metadata layer for strong consistency, high‑performance metadata, multi‑protocol support (POSIX, HDFS, S3), small‑file management, and deep Kubernetes integration.

The architecture follows the classic GFS/HDFS three‑tier model (metadata, data, client). Data is stored on any object storage service, while the metadata engine currently uses Redis (with plans for additional engines such as MySQL, TiKV, etc.). The client implements full POSIX semantics, random read/write, and other advanced features.

Typical cloud‑native workloads—Hadoop ecosystem, AI training pipelines, and Kubernetes‑based services—benefit from JuiceFS’s ability to provide a single, high‑performance, POSIX‑compatible storage layer that eliminates data movement and simplifies operations.

Observability is addressed through built‑in logging and analysis tools that expose detailed per‑API latency and access patterns, helping users pinpoint performance bottlenecks.

The speaker concludes with a roadmap that includes expanding metadata engine support, further performance optimizations, and broader ecosystem integrations.