Applying Alluxio to Autonomous Driving Model Training: Deployment, Performance, and Operational Insights

In autonomous driving model training, a data closed‑loop is built where massive sensor data (camera images, LiDAR point clouds) are collected, stored, parsed, labeled, and used for training tasks such as object detection and lane detection, followed by simulation validation and deployment.

The previous NAS‑based solution suffered from poor concurrency when many training jobs ran simultaneously, difficult management due to scattered scripts and directories, significant space waste from duplicated data, and complex usage because each dataset required custom download logic.

Alluxio addresses these issues by providing a unified, distributed cache that improves concurrency, automatically evicts data via LRU policies, eliminates duplicate storage through a shared namespace, and simplifies access via a FUSE interface, reducing both operational overhead and risk of accidental deletions.

Deployment was carried out first in a single‑datacenter setup colocated with GPU nodes, using local SSDs, with FUSE clients and worker nodes forming a small cache cluster that communicates with underlying object storage; a multi‑datacenter deployment replicated this architecture in each site, ensuring consistent bucket naming and handling separate S3 endpoints.

Functional testing verified that the existing training pipeline could be minimally modified to work with Alluxio, covering PVC configuration in Kubernetes, dataset organization, job submission settings, path replacement scripts, and final access APIs.

Performance testing showed that on a single node Alluxio matched NAS throughput, while on multiple nodes Alluxio scaled dramatically, reaching over 20 GB/s and maintaining stable performance compared to NAS, which plateaued around 7–8 GB/s with high variance.

Operational tuning involved expanding ETCD from one to three nodes for high availability, adjusting S3 path handling to satisfy security policies, configuring FUSE concurrency limits and direct‑memory allocation, and maintaining detailed incident logs, operation manuals, and version‑controlled configuration files.

Additional requirements from R&D and operations included ensuring stability (preventing FUSE crashes), determinism (predictable preload times), controllability (manual cache eviction via file lists), a configuration center for change impact analysis, end‑to‑end latency tracing across FUSE, workers, and UFS, and intelligent monitoring to detect emerging issues automatically.

Overall, Alluxio significantly improves usability, reduces storage costs by eliminating 20‑30% redundant data, automates data cleanup, and accelerates training by up to tenfold, making the autonomous driving data pipeline more efficient and maintainable.