Xiaomi Data Management and Application Practice: Metadata Platform Construction, Applications, and Future Plans

In this talk, Xiaomi shares its experience in data management and application, focusing on the construction of a metadata platform that supports higher‑level services like data maps, governance, cost control, and quality improvement.

The metadata platform is built around three key improvements: achieving full‑domain metadata, real‑time lineage, and precise measurement. Full‑domain metadata expands beyond Hive to include Talos, Doris, Kudu, Iceberg, ES, MySQL and other engines, providing a unified view of data assets.

Technical architecture mirrors Apache Atlas and consists of three layers: a source layer (Metadata Source, Lineage Source, Log Source, Application), an integration layer (Metacat, MQ, API), and a storage layer (MySQL for basic info, Hive for snapshots, JanusGraph for lineage, Elasticsearch for search and audit).

Real‑time lineage is realized by embedding MQ hooks in engines such as Hive, Flink, Spark, and Presto, combined with SQL proxy logs to accurately capture data flow.

Precise measurement aligns data usage statistics with HDFS‑Image storage metrics and SQL audit logs, enabling accurate access counts and cost accounting.

Based on the platform, Xiaomi develops four main metadata applications: data maps (search and lineage), data governance (modeling compliance and completeness), data cost governance (cost analysis loop, unified storage measurement, daily billing, user attribution, and real‑time cost estimation), and data quality (timeliness and content checks, with dimensions of uniqueness, accuracy, completeness, and consistency).

Cost governance has delivered a 40% reduction in data expenses, demonstrated by a comparison of historical, projected, and actual cost curves.

Future plans include tighter production‑guarantee resource scheduling, a long‑term roadmap for metadata health definition and remediation, and business enablement through improved quality, efficiency, and cost visibility.

Overall, the presentation outlines how a robust metadata platform can drive data asset visibility, governance, and optimization across a large‑scale organization.