How Fluid’s Cloud‑Native Caching Supercharges AIGC Model Inference

Challenges of Large‑Model Inference

Cost, performance, and efficiency are the three core factors that affect the production and use of large AI models. As model size grows, GPU resources become scarce and expensive, making per‑inference cost a primary concern. Performance determines user stickiness, while engineering efficiency dictates how quickly models can be updated.

In AIGC scenarios, the separation of compute and storage leads to high latency and limited bandwidth, exacerbating cost and performance problems. For example, loading a 340 GiB Bloom‑175B model from object storage takes about 71 minutes (≈85% of total startup time) with an I/O throughput of only a few hundred MB/s.

Why Cloud‑Native Kubernetes Is Essential

Kubernetes standardizes heterogeneous resources, simplifies operations, and leverages elasticity to reduce GPU‑related costs. It has become the de‑facto runtime for AI workloads, enabling shared or dedicated model serving, edge, serverless, and multi‑cluster deployments.

Fluid Project Overview

Fluid is an open‑source system that orchestrates data and compute tasks on Kubernetes. It provides:

Standardized data‑access patterns and cache orchestration (e.g., Alluxio, JuiceFS, JindoFS, EFC).

CRD‑based automation for data pre‑heating, migration, and cache scaling.

Accelerated performance through cache‑aware scheduling and task‑data affinity.

Runtime‑agnostic deployment across native, edge, serverless, and multi‑cluster Kubernetes.

End‑to‑end data‑flow orchestration that automates model‑cache preparation, deployment, and cleanup.

Fluid defines two CRDs: Dataset (describes the model data source) and Runtime (represents the chosen cache system). Creating these resources automatically provisions PVCs, launches cache components, and makes the cached data available to inference pods.

Elastic Distributed Cache and Scaling

Fluid’s elastic cache converts limited external bandwidth into scalable intra‑cluster bandwidth by adding cache worker nodes. Tests show near‑linear bandwidth increase as the number of cache nodes grows, reducing model startup time from hours to minutes.

Elastic scaling also enables cost‑effective operation: cache can be expanded during traffic spikes and shrunk to zero when idle, matching the I/O patterns of large‑language models or image‑generation workloads.

Performance Evaluation

Using HuggingFace Text‑Generation‑Inference with a 12.55 GB model stored in OSS, direct OSS access required ~101 seconds for pod readiness. After deploying Fluid’s cache (≈40 seconds to provision) and adding a pre‑heat annotation, the same deployment became ready in 22 seconds, a ten‑fold speedup. A second pod started in only 10 seconds because the data was fully cached.

Further optimization with Fluid’s Python SDK (multithreaded reads and pre‑fetch) halved cold‑start latency, enabling a ~100 GB model to load in under one minute.

Key Takeaways

Fluid delivers an out‑of‑the‑box, cloud‑native solution that improves AIGC inference performance, reduces GPU‑related costs, and provides end‑to‑end automation for data preparation, cache management, and deployment scaling.