Why Metadata Governance Is the Backbone of Modern Data Platforms
This article explains how metadata serves as essential infrastructure for data platforms, detailing Huawei's classification framework, governance challenges, management architecture, integrated modeling, data lake handling, service management, and data map construction to bridge business and IT domains.
1. Huawei Data Classification Management Framework
Huawei classifies data into internal and external, structured and unstructured, and metadata, establishing a comprehensive classification framework illustrated in the diagram.
2. Challenges of Metadata Governance
Before governance, data was often "unfindable, unreadable, and untrustworthy," leading to scenarios where analysts struggled to locate or understand data across dozens of IT systems, faced complex storage structures, and spent excessive time on manual conversion and verification.
Key pain points include the disconnect between business and technical metadata and the lack of efficient data search tools for business users.
3. Metadata Management Architecture and Strategies
The architecture covers four stages: metadata generation, collection, registration, and operation.
Generation: Define processes and standards to link business and technical metadata during IT product development.
Collection: Automatically gather metadata from various IT systems using a unified meta‑model.
Registration: Apply incremental and stock‑based methods to register foundational metadata.
Operation: Build a company‑wide metadata center to manage the full lifecycle.
Huawei also establishes unified management methods, platforms, and standards to ensure data “enters the lake with justification and can be retrieved” and to enable data map creation.
4. Integrated Modeling Management
Huawei addresses the historical gap between information architecture and IT implementation by promoting integrated modeling, which aligns metadata verification, publishing, and registration, and supports continuous product data model operation and physical model standardization.
5. Metadata and Data Lake Management
Huawei’s data lake aggregates structured and unstructured raw data, governed by a unified semantic layer. Data entry follows six standards, ensuring ownership, data standards, security level, source clarity, quality assessment, and metadata registration.
Both structured and unstructured data have specific ingestion processes, with unstructured data requiring basic feature metadata, content parsing, relationship mapping, or raw file storage.
6. Metadata and Data Service Management
Standardized data service identification clarifies service types, reuse potential, and admission criteria, reducing redundant development and enhancing service reusability.
7. Metadata and Data Map Construction
To improve data discoverability and understandability, Huawei builds a data map that visualizes sources, quantity, quality, distribution, standards, and relationships, enabling users to efficiently locate and comprehend data for consumption.
Data Thinking Notes
Sharing insights on data architecture, governance, and middle platforms, exploring AI in data, and linking data with business scenarios.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.