How Fluid Enables Seamless Dynamic Dataset Mounting for Cloud‑Native AI Development

Background

Cloud‑native technologies improve AI development efficiency through flexible resource management and standardized operations. The best‑practice called CNAI (Cloud Native AI) solves software environment maintenance, heterogeneous hardware management, and uneven resource allocation, allowing data scientists and AI developers to focus on core business logic.

PAI‑DSW (Data Science Workshop)

PAI‑DSW is a one‑stop cloud‑native AI development platform built on Alibaba Cloud Kubernetes. It provides JupyterLab, WebIDE, and Terminal environments, runs entirely on containerized design atop Alibaba Cloud Container Service (ACK), supports mounting OSS/NAS/CPFS datasets, and offers lifecycle management and VPC network integration.

Challenges in Cloud‑Native AI Development

In Kubernetes, data operations lack flexibility. The CSI interface does not define how applications manage heterogeneous data sources, version datasets, or dynamically update mount points, creating a gap for AI developers who need to add or remove datasets without restarting pods.

Fluid Overview

Fluid is a Kubernetes‑native distributed dataset orchestration and acceleration engine. It abstracts data as “elastic datasets” that can be dynamically mounted, providing high‑performance access for AI and big‑data workloads. Fluid introduces a mutable API, asynchronous tuning, and a mount‑point propagation mechanism to enable dynamic dataset changes without pod restarts.

Fluid Architecture

Fluid consists of a control plane and a data plane. The control plane includes Fluid Dataset, ThinRuntime/ThinRuntimeProfile, Fluid Controller, and Fluid Webhook. The data plane includes a FUSE pod (or sidecar) running the Fluid DynamicMountEngine, which reacts to dataset updates, creates or removes FUSE mount processes, and exposes mount status metrics.

Key components:

Fluid Dataset : Logical collection of one or more data sources, supporting mutable API for dynamic updates.

ThinRuntime and ThinRuntimeProfile : Define custom storage system integration and resource limits.

Fluid Controller : Watches Dataset changes and updates ConfigMaps for the data plane.

Fluid Webhook : Injects sidecar containers in serverless pods to handle mount propagation.

Dynamic Mount Implementation in PAI‑DSW

PAI‑DSW uses Fluid to enable dynamic mounting of OSS datasets. The workflow includes creating Fluid CRs (Dataset, ThinRuntime, ThinRuntimeProfile), generating Secrets for temporary OSS tokens, and configuring PVCs that reference the Fluid Dataset. When a user adds a new dataset, Fluid updates the mount points in the running pod without restarting.

apiVersion: data.fluid.io/v1alpha1
kind: Dataset
metadata:
  name: demo-for-dynamic-mount
spec:
  mounts:
  - mountPoint: oss://bucket-name-1
    name: mount-point-1
    path: /subpath-for-mount-point-1
    options:
      fs.oss.endpoint: oss-cn-shanghai-internal.aliyuncs.com
  - mountPoint: oss://bucket-name-2
    name: mount-point-2
    path: /subpath-for-mount-point-2
    options:
      fs.oss.endpoint: oss-cn-shanghai-internal.aliyuncs.com

apiVersion: data.fluid.io/v1alpha1
kind: ThinRuntime
metadata:
  name: demo-for-dynamic-mount
spec:
  profileName: profile-demo-for-dynamic-mount
---
apiVersion: data.fluid.io/v1alpha1
kind: ThinRuntimeProfile
metadata:
  name: profile-demo-for-dynamic-mount
spec:
  fileSystemType: thin
  fuse:
    image: xxx
    imageTag: xxx
    command:
    - tini
    - -g
    - --
    - /usr/local/bin/entrypoint.sh

Using Dynamic Mounts

After starting a DSW instance, developers can mount a new OSS dataset with two lines of Python code:

from pai.dsw import mount
mount("oss://bucketname/xxxxx")

They can list current mounts:

from pai.dsw import list_dataset_configs
list_dataset_configs()

And unmount when finished:

from pai.dsw import unmount
mount_path = '/mnt/dynamic/your_mount_path/'
unmount(mount_path)

Future Outlook

Current version supports dynamic mounting of OSS datasets. Future work includes supporting more storage types, improving secure and performance‑preserving mount propagation, adding cache systems for shared datasets, and exploring writable dataset scenarios.