How Fluid Enables Cloud‑Native Elastic Data for AI Workloads

Introduction

Kubernetes provides low‑level storage interfaces such as CSI, but it does not define how applications should efficiently use and manage data in container clusters. AI workloads, in particular, need higher‑level data access and management capabilities.

Challenges for AI Data Access

Closed‑loop serverless data access: Serverless compute is popular for AI because of elasticity and cost savings, yet its data access is often tied to vendor‑specific object stores, making third‑party or self‑built storage hard to integrate via standard interfaces.

Lack of standardized hybrid‑cloud data access: AI workloads require high‑performance access to heterogeneous sources (HDFS, S3, Lustre, etc.) across hybrid clouds, but the community has not defined a cloud‑native distributed cache abstraction.

Dynamic data source switching barriers: Data scientists frequently need to switch datasets during development, but persistent volumes are immutable, forcing pod recreation and slowing iteration.

These gaps represent missing capabilities in the cloud‑native container ecosystem.

What is Fluid?

Fluid implements the “cloud‑native elastic data abstraction” as a first‑class citizen in Kubernetes, forming the basis of a cloud‑native data orchestration and acceleration system.

Key Concepts

Dataset: A mutable data abstraction that lets users define a set of heterogeneous data sources.

Runtime: A pluggable distributed cache system.

DataOperation: Enables proactive data pre‑warming, migration, and processing.

Vision

Fluid aims for Data Anyway (simple access), Data Anywhere (run everywhere), and Data Anytime (on‑demand usage).

Data Anyway: Focuses on easy data access and diverse compute resources.

Architecture

Fluid supports both serverful (traditional) and serverless compute:

Serverful uses CSI mode with standard PVC/PV mounting.

Serverless uses Sidecar mode for lightweight, “process‑transparent” access.

Storage vendors can integrate via a simplified plugin model without deep CSI knowledge. Fluid also provides a universal CacheRuntime CRD that abstracts various cache engines (Alluxio, JuiceFS, Vineyard, JindoFS, EFC, etc.) and storage types (CPFS, NAS, OSS, OSS‑HDFS, Cubefs, GlusterFS, and custom solutions).

Dynamic dataset mutability allows runtime updates of data sources without pod recreation, greatly improving developer productivity.

Community Updates

Fluid was accepted as a CNCF Sandbox project in May 2021 and has grown to over 500 contributors and 40 production adopters.

CNCF 2024 Technology Radar Recognition

At KubeCon North America 2024, CNCF listed Fluid in the “Adopt” quadrant of its Technology Landscape Radar, highlighting its maturity and strong developer recommendation in batch, AI, and ML workloads.

The radar notes that Apache Airflow, CubeFS, Kubeflow, and Fluid are all placed in the “Adopt” category, indicating broad production use.

Join the Fluid Community

Project repository: https://github.com/fluid-cloudnative/fluid

Slack: #fluid channel (CNCF workspace)

We welcome new contributors and users to join the community.