Alluxio for AI and Machine Learning: Architecture, Optimizations, and Performance Evaluation

The session, hosted by DataFunTalk, features speakers Chen Shouwei (Rochester University PhD, Alluxio core engineer) and Mao Baolong (Tencent Alluxio OTeam lead), who introduce Alluxio and its support for AI scenarios, followed by detailed discussion of business background, development, tuning, benchmark comparisons, and future work.

Alluxio Overview

Alluxio is the world’s first distributed ultra‑large‑scale data orchestration system, originated from UC Berkeley AMP Lab and now an open‑source project with over 300 organizations and 1,100 contributors. It is deployed in production by eight of the top ten internet companies worldwide.

Alluxio in AI Workloads

In the big‑data ecosystem, Alluxio sits between AI/ML compute frameworks (Spark, Presto, TensorFlow) and distributed storage systems (e.g., Ceph). It provides a unified client, API, and global namespace, enabling feature‑calculation jobs that read many small files with high concurrency to benefit from in‑memory caching and reduced storage latency.

Core Functions

Distributed cache that brings data close to compute nodes, improving data locality for Spark, Presto, TensorFlow, etc.

Multiple APIs (HDFS, POSIX, etc.) allowing seamless migration of existing workloads.

Unified namespace that abstracts underlying storage, simplifying data access across heterogeneous back‑ends.

Challenges in Machine‑Learning Scenarios

Training data consists of millions of tiny files (e.g., images) rather than a few large files.

Dataset size exceeds the capacity of a single node, requiring distributed storage.

High‑concurrency distributed training demands very high I/O throughput.

Optimization Strategies

1. Metadata management : choose Heap metastore for up to tens of millions of files, or RocksDB off‑heap for billions of files. Adjust Java heap/off‑heap settings accordingly.

ALLUXIO_MASTER_JAVA_OPTS+=
" -Xms256g -Xmx256g "
ALLUXIO_JAVA_OPTS+=
"-XX:MaxDirectMemorySize=10g "

2. Reduce client/worker connection buffers to support thousands of concurrent threads.

3. Tune JVM GC parameters based on observed GC frequency.

4. Cache metadata on the master side (active sync interval, e.g., 30 s) and optionally extend the interval for less‑dynamic workloads.

alluxio.master.ufs.active.sync.interval=
30s
ec

5. Enable Fuse‑side metadata cache (attr_timeout, entry_timeout) to avoid round‑trips to the master.

6. Disable audit logging in pure‑read scenarios to eliminate a major throughput bottleneck.

alluxio.master.audit.logging.enabled=
false

7. Adjust dynamic configuration via the new updateConf API, allowing on‑the‑fly parameter changes without stopping the service.

StressBench – Alluxio Performance Test Tool

An open‑source stress testing tool bundled with Alluxio that can generate read/write and metadata workloads against a running cluster. Example usage with the Fuse client demonstrates single‑node throughput of 183 MiB/s.

Business Background – Game AI

Feature‑calculation for a MOBA game involves both supervised and reinforcement learning pipelines. Game binaries (gamecore) are stored on CephFS; Alluxio is deployed on a Kubernetes‑based compute platform, with 1,000 workers (~4 TB) and 1,000 pods (each 4 CPU) handling 250 k game matches.

Ratis‑Shell for HA Management

A custom shell built on top of Apache Ratis enables leader switching and status queries for Alluxio HA clusters. The project is open‑sourced (github.com/opendataio/ratis-shell) and shared with the Ozone community.

Observability, Logging, and Usability

More than 20 monitoring metrics (block count, RPC ops, Ratis stats, JVM/GC, cache hit‑rate, etc.) have been added. Centralized logging and alerting provide full visibility. Improvements include distributedLoad pre‑warm, fixing OOM in Job Service, visualizing mount tables, and supporting IP/hostname communication.

Future Work and Open‑Source Contributions

Plans include raising throughput limits, decoupling Alluxio FUSE via CSI, enhancing local‑cache and metadata‑cache, load‑balancing, and intelligent read/write scheduling. Tencent contributes over 20 developers, 1 PMC, 2 committers, >400 merged PRs, and numerous blog posts and webinars to the Alluxio community.

For more details, see the Alluxio whitepaper (https://www.alluxio.io/resources/whitepapers/alluxio-for-machine-learning-deep-learning-in-the-cloud/) and related GitHub repositories.