Douyin’s Data Asset Platform: Transforming Big Data Lineage

Overall Overview

Douyin Group’s one‑stop Data Asset Management Platform redefines data governance by focusing on "data assets" rather than just raw metadata. It integrates extensive data source types, collects metadata into a unified lake, and provides full‑linkage lineage, enabling systematic management, classification, and enrichment of assets.

The platform supports secondary operations such as asset onboarding/offboarding, grading, and classification, and leverages proactive metadata techniques to enrich asset information. An asset evaluation system continuously assesses completeness, while consumption capabilities include search, portal, recommendation, and AI‑driven search powered by large language models.

Focus on Full‑Linkage Data Lineage

The presentation concentrates on four aspects: an overall introduction, system architecture, application scenarios, and future outlook of the lineage component.

Construction Background

See the chain: With millions of tasks across the group, lineage reveals relationships between business processes.

Ensure quality: Real‑time lineage helps assess the impact of frequent task changes on production stability.

Guarantee security: Lineage tracing is essential for discovering and protecting sensitive data.

Reduce cost: Accurate lineage enables efficient resource utilization and identification of low‑value assets for governance.

Building comprehensive, real‑time, and accurate big‑data lineage is therefore urgent.