Boosting Cloud‑Native AI Training with Alluxio: Performance Tuning on Kubernetes

Background

Deep‑learning has become a dominant AI technology, driving massive demand for efficient model training. In cloud environments, container orchestration platforms such as Docker and Kubernetes provide elastic compute resources, while object storage offers cheap, scalable data storage. However, the compute‑storage separation architecture introduces significant data‑access latency, especially when training large models on GPU clusters.

Typical Data‑Access Challenges

High synchronization cost : Frequent data updates require costly sync operations.

Expensive cloud storage : Paying for high‑performance distributed storage adds overhead.

Scaling difficulty : Replicating terabytes of training data to each node becomes time‑consuming.

Proposed Architecture

The solution combines Kubernetes, Kubeflow, and Alluxio to create a container‑based, data‑orchestrated training platform. Alluxio acts as a distributed virtual file system that provides a unified namespace, hierarchical caching, and multiple access interfaces, enabling efficient data reads from private data centers or public object storage.

Core Components

Kubernetes : Manages container clusters, supporting CPU, GPU, and NPU instances on Alibaba Cloud (ACK).

Kubeflow : Cloud‑native AI platform that schedules distributed TensorFlow training (parameter‑server and AllReduce modes) via the Arena extension.

Alluxio : Provides a caching layer between storage and compute, exposing a POSIX‑compatible FUSE interface for AI workloads.

Experimental Setup

We trained a ResNet‑50 model on the ImageNet dataset (144 GB, TFRecord format) using four V100 nodes (32 GPUs). Data resided in Alibaba Cloud Object Storage; Alluxio cached data in memory (40 GB per node, total 160 GB) without pre‑loading.

Performance Bottlenecks

When upgrading from P100 to V100 GPUs, compute speed increased >3×, exposing I/O limits. With eight GPUs, Alluxio‑backed training achieved only ~30 % of the synthetic‑data baseline, and system metrics showed no CPU, memory, or network saturation, indicating I/O inefficiencies in the Alluxio‑FUSE stack.

Root‑Cause Analysis

Alluxio’s distributed file system incurs multiple RPCs per read, adding latency in high‑throughput training.

Cache eviction and data‑shuffling cause additional overhead.

FUSE reads are limited to 128 KB per call and spawn many threads, leading to CPU waste.

Kubernetes limits container CPU shares (default 2), causing Java’s availableProcessors() to report only one core, throttling Alluxio client concurrency.

Optimization Strategies

FUSE Tuning

Upgrade Linux kernel (e.g., 4.19) to benefit from FUSE improvements (+20 % read performance).

Extend metadata TTL with -o entry_timeout=T -o attr_timeout=T.

Configure max_idle_threads to match the number of I/O threads and patch libfuse2 to support this option.

Alluxio Configuration

Set alluxio.user.ufs.block.read.location.policy to LocalFirstAvoidEvictionPolicy and reserve cache space via alluxio.user.block.avoid.eviction.policy.reserved.size.bytes.

Disable passive local caching with alluxio.user.file.passive.cache.enabled=false.

Change default read type from CACHE_PROMOTE to CACHE to avoid costly block moves.

Enable metadata caching: alluxio.user.metadata.cache.enabled=true and tune size/expiration.

Increase worker list refresh interval and optionally disable last‑access‑time updates.

Data Locality

Prefer direct file access over Unix‑socket mode to reduce network hops, ensuring Alluxio Worker and Alluxio‑FUSE share the same hostname/IP and cache directory.

Java & Kubernetes Adjustments

Set JVM -XX:ActiveProcessorCount (or -XX:ParallelGCThreads, -XX:ConcGCThreads, -XX:CICompilerCount) to reflect actual CPU resources.

Allocate sufficient CPU requests/limits in pod specs to avoid the default cpu_shares of 2.

Results

After applying the optimizations, ResNet‑50 training on an 8‑GPU node achieved a 236 % speedup (31 068.8 images/s vs 12 044.8 images/s baseline) and scaled efficiently to four nodes (8 GPUs each) with only a 3.29 % performance loss compared to synthetic data. Compared with cloud SSD storage, Alluxio delivered a 70.1 % throughput increase. Training time dropped from 110 minutes (SSD) to 65 minutes, saving 45 minutes and reducing cost by ~40 %.

Conclusion & Future Work

The study identifies key bottlenecks of Alluxio‑FUSE in high‑concurrency deep‑learning scenarios and demonstrates practical configuration and system‑level tweaks that substantially improve performance. Future work includes enhancing page‑cache support, stabilizing FUSE under massive small‑file workloads, and continued collaboration between Alibaba Cloud, the Alluxio community, and academic partners.