How Douyin’s Data Asset Platform Revolutionizes Big Data Lineage

This article briefly introduces Douyin Group’s Data Asset Management Platform, a new direction for handling complex business scenarios that encourages fresh thinking about metadata and data assets. It emphasizes the evolution and application of Douyin’s big data lineage, providing a macro perspective and constructive ideas for building robust lineage.

Unlike the industry focus on pure metadata, Douyin’s platform centers on "data assets" to meet precise data discovery needs. By integrating powerful metadata collection, the platform aggregates all source metadata into a unified metadata lake, including full‑link lineage. Data business partners then perform secondary management such as publishing, classification, and asset evaluation, while AI‑driven search and recommendation capabilities enable diverse consumption of data assets.

The presentation focuses on full‑link lineage and is organized around four topics:

Overall introduction of Douyin Group’s lineage

System architecture of the lineage platform

Application scenarios of the lineage

Future outlook

The primary goal of building data lineage at Douyin is to achieve comprehensive, real‑time, and accurate big data lineage, enabling scenario‑wide applications that improve efficiency.

Construction background highlights four key motivations:

Visibility: With millions of tasks, lineage clarifies relationships across the data chain.

Quality assurance: Real‑time lineage helps assess the impact of frequent task changes on production.

Security: Lineage aids in discovering and protecting sensitive data.

Cost reduction: Accurate lineage enables optimal resource utilization and low‑value asset governance.

Thus, establishing robust big data lineage is an urgent priority for Douyin.

Article excerpted from "A Plain‑spoken Big Data e‑Book" (first chapter).