Tencent's Data Governance Practices and Technical Implementation

The presentation begins with an overview of data governance, defining it as a collection of organizational structures and activities that guide data management, emphasizing goals such as data sharing, quality improvement, asset valuation, and compliance.

It then outlines the difficulties faced in data governance, including heterogeneous data sources, lack of standards, long data pipelines, and security and compliance concerns.

Tencent's approach follows a "know‑govern‑manage" methodology: first identifying data objects and their locations, then applying standardization and governance techniques, and finally continuously improving data quality through feedback loops.

The Tencent data governance system is organized into a unified OTeam that coordinates governance and security across the company, establishes enterprise‑level standards, and provides a one‑stop platform for data governance tools, community operations, and awareness.

The business framework consists of four pillars—asset creation, assessment, operation, and control—supporting a full lifecycle from metadata collection and storage to lineage tracking and quality reporting, with both offline (HBase, ES, graph DB) and online (relational DB) storage choices.

Metadata management includes asset ownership clarification, value analysis based on business attributes and access frequency, data cleaning with Redis‑based diff detection, and lifecycle recommendations aided by lineage and usage patterns.

Security governance is addressed through data classification, tiered access controls, encryption, masking, and audit logging, with a maturity model spanning five levels of capability.

Technical practice details the unified metadata service architecture, divided into data‑governance and platform‑service layers, and the microservice decomposition that supports metadata products, computation engines, and data sources.

Data collection employs both scheduled (incremental and full) and real‑time pipelines, using Redis to filter unchanged data and GUID generation to achieve idempotent upserts across heterogeneous storage systems.

Lineage analysis captures SQL execution plans from engines like Hive and Spark, parses them with a Druid‑based parser, and stores relationships in a graph database, with optimizations to avoid ambiguous task‑node mappings.

Overall, the article shares Tencent's experiences, challenges, and solutions in building a large‑scale data governance platform, offering guidance for practitioners facing similar data management problems.