How Douyin’s Data Platform Overcomes EB‑Scale Metric Challenges

Data Challenges

Douyin Group, the core of ByteDance’s domestic information and service business, runs apps such as Douyin, Toutiao, and Xigua Video, handling data volumes that exceeded the exabyte level years ago, creating huge data‑management challenges.

Volcano Engine Intelligent Platform

The platform serves as an agile data‑intelligence engine and is divided into four layers: data engine, data construction & management, data analysis applications, and solution & consulting services, covering the entire data lifecycle from collection to analysis.

The data products focus on two core traits: agility and ease of use. Speed and flexibility are achieved through real‑time data pipelines capable of sub‑second queries on billion‑row datasets. Ease of use is ensured by low‑threshold, no‑code tools that let non‑technical users build data portals and run A/B tests, with seamless integration to Feishu, calendars, and business systems.

Metric Construction Pain Points

Three major difficulties arise as the business matures:

Inconsistent metric management: “Same name, different meaning” or “different name, same meaning” make pure documentation insufficient.

Unclear metric definitions: Business users often face ambiguous metric definitions and technical specifications.

Fragmented metric consumption: Early metric definitions lead to redundant tables and duplicated metrics when new business scenarios appear.

Overall Technical Solution

A three‑layer solution is proposed:

Metric production – model design and data‑quality guarantees.

Metric management – improving efficiency and consistency.

Metric consumption – delivering metrics as services via topic‑based consumption.

Metric Management Practice

Key issues include metric consistency (duplicate names or conflicting values), continuous freshness and effective iteration, and efficiency in definition and breakdown. The team builds a closed loop where production drives consumption and feedback improves management.

Collaboration Process

1. Requirement registration by analysts. 2. Metric check and decomposition by data‑warehouse developers. 3. Model creation and binding by infrastructure teams. 4. Delivery to product teams or analysts with documentation. 5. Full‑link tracing of model binding, metric classification, and consumption.

Organizational Design for Accountability

Roles are clearly defined:

Business owners: Define metric requirements and hold final interpretation rights.

Data application layer: Transform metric models into data products, ensuring alignment with business definitions.

Public‑layer data team: Maintain foundational information and ensure consistency.

Ensuring Metric Consistency

Two aspects are emphasized:

Standardized metric decomposition covering domain, process, measure, modifiers, granularity, period, unit, and type.

Unique‑metric validation and similarity checks for atomic metrics and modifiers.

Improving Metric Decomposition Efficiency

Strategies include:

Conceptual innovation: Focus on core metrics first, adopt “develop‑then‑decompose” for fast‑changing short‑video business.

Process optimization: Document business processes, atomic metrics, and modifiers; create operation manuals and metric trees; automate repetitive tasks with scripts.

Exploring Large‑Model Automatic Decomposition

The team experiments with large‑model AI to automatically split tables and fields into atomic metrics, modifiers, and periods.

Metric Production Practice

Model design follows a hierarchical approach from detail to summary layers (detail, light‑aggregation, coarse‑aggregation). Light‑aggregation aims to cover many dimensions while balancing performance; coarse‑aggregation focuses on downstream consumption, typically using three key dimensions.

Quality Assurance for Metric Production

Key steps include product module mapping, responsibility allocation, and setting concrete standards for timeliness and user experience. Implementation relies on full‑link lineage analysis, daily alerting via Feishu, and rigorous testing during development, release, and operation phases.

Metric Consumption Practice

Metrics are organized into official metric topics—virtual tables that hide physical implementation. Advantages include low construction cost, fast metric discovery, cross‑cluster/source analysis, and intelligent routing to the most stable model.

Topic management offers flexible directory structures, fine‑grained permission control, easy import, and a one‑stop view of metric definitions, technical details, and consumption lineage.

Applications include decision dashboards, self‑service data retrieval via Chat‑BI, and automated daily reports.

Summary and Outlook

The platform aims to provide stable, reusable data products through standardized, configurable, and automated metric production, AI‑assisted metric management, and unified metric consumption. Future work will focus on scaling these three core areas to further improve efficiency, transparency, and data‑driven decision making.