Xiaomi AI Data Management Platform: Design, Implementation, and Practice

The article shares the background, design scheme, and practical implementation of Xiaomi's AI Data Management Platform.

AI data, encompassing text, images, video, audio, and sensor data, is a core component of AI infrastructure and can be categorized as structured, semi‑structured, or unstructured, with non‑tabular data accounting for the majority of volume.

Two main drivers prompted the platform's construction: the external AI boom demanding efficient, secure, and intelligent data handling, and internal challenges such as privacy risks, low data‑use efficiency, asset management difficulties, lack of debugging environments, and fragmented AI‑Data workflows.

Inspired by industry solutions like Databricks' Unity Catalog and Snowflake's Fileset, Xiaomi introduced a unified catalog (MetaCat) for tables and a Fileset concept for non‑tabular data, enabling consistent permission auditing and cross‑source governance.

The solution integrates all data into a single metadata system, treating Fileset as a special table that can be accessed via SQL, Python, or Scala, and provides notebook‑based interactive development for algorithm engineers.

Four core capabilities were deployed: seamless integration of non‑tabular data, notebook development environment, data lineage for governance, and comprehensive asset management (cost, permission, lifecycle). These yielded reduced workflow complexity, up to 80% storage cost savings with the LavaFS system, and improved data traceability.

The summary outlines future plans to extend Fileset to additional storage backends (e.g., JuiceFS), support major ML frameworks (PyTorch, TensorFlow), connect AI application and resource platforms, and continuously enhance the product experience.

The Q&A section addresses motivations, definitions of non‑tabular data, cost considerations, version management for model files, and storage strategies.