Douyin Group Data Asset Management Platform and Data Lineage Architecture Overview

The article introduces Douyin Group's one‑stop data asset portal, emphasizing the shift from traditional metadata to a systematic "manage, find, use" data asset platform that supports diverse data sources and provides unified metadata lakes with full‑link lineage.

The platform enables asset evaluation, search, recommendation, and AI‑driven discovery, facilitating data consumption through portals and productized capabilities.

Four key topics are covered: overall data lineage overview, system architecture, application scenarios, and future outlook.

1. Data Lineage Overview – Douyin aims to build comprehensive, real‑time, accurate big‑data lineage to empower various scenarios, recognizing lineage as the core of metadata.

The lineage covers source‑level, production‑level (real‑time and offline warehouses), and application‑level relationships, with metrics such as coverage, accuracy, and completeness.

2. System Architecture – Challenges include fine‑grained parsing, unstructured sources, cross‑region coverage, and large‑scale application tracing. The solution comprises data source ingestion, metadata and lineage collection, storage using graph databases (JanusGraph, Neo4j, NebulaGraph), and analysis services.

Parsing leverages Antlr and Calcite to handle multiple dialects and complex scripts, converting syntax trees into lineage information.

Three lineage services are described: production lineage, cross‑region lineage, and application lineage, each addressing specific data flow and tracing needs.

3. Application Scenarios – Includes data development (impact assessment, field lineage, task acceleration), data governance (risk asset identification, cost calculation, timeliness and accuracy assurance, security), and data asset utilization.

Use cases demonstrate how lineage supports cost reduction, quality improvement, and efficient data operations.

4. Future Outlook – Plans focus on full coverage, standardized lineage APIs, open contributions, finer granularity (row‑level lineage), and deeper exploitation of lineage for quality, efficiency, and security.

Douyin's DataLeap platform aims to continuously enhance metadata, integrate active metadata and large‑model capabilities, and expand to the broader market.