How Bilibili Transformed Big Data Governance: From Reactive Storage Management to Proactive Multi‑Dimensional Control

Bilibili Big Data History

Internet companies face exploding data volumes as they scale, requiring robust big data capabilities. Bilibili, founded in 2009, built its big data team only in 2017 and began large‑scale construction in 2019. By 2023 the platform handled exabyte‑scale data, making the traditional approach of purchasing servers unsustainable.

Challenges of Rapid Data Growth

Multiple business lines manage data independently without a unified system.

Business and technical data are mixed.

Lack of asset ownership leads to orphaned data.

No unified standards for data reporting.

To address these, Bilibili initiated the "Wanglou" (Watchtower) project in late 2021, aiming to build a comprehensive data‑asset governance framework.

Problem 1: How to Start Governance?

The project began by defining asset metadata and a governance indicator system. First, a bottom‑up approach identified key data assets (Hive tables, scheduled jobs) and mapped their lifecycle processes. Later, a top‑down approach used the indicator system to guide governance actions.

Asset Metadata Model

Asset metadata provides a high‑level view of data assets, supporting inventory, problem discovery, and strategy formulation.

Governance Indicator Framework

The framework consists of governance goals, strategies, and evaluation metrics. Goals define target KPIs (e.g., storage reduction), strategies break goals into actionable items, and evaluation metrics measure implementation effectiveness.

Storage Governance as the First Focus

Storage watermarks frequently exceeded 90%, causing emergency clean‑ups. By analyzing top‑consuming assets, Bilibili identified common issues such as unused downstream tables, overly long TTLs, and uncompressed data. Standard definitions and strategies were created to address each problem.

Standard‑Strategy Table

Problem

Standard

Strategy

Unused downstream

Offline low‑heat data

Push offline

TTL too long

Set TTL by timeliness or tier

Shorten TTL

Uncompressed data

Require compression

Enforce compression

Scoring Model and Cost Transparency

Bilibili introduced a billing system that breaks down costs by asset type (storage, compute, traffic) and publishes departmental, space‑level, and personal bills. A scoring model converts governance actions into points, reflecting compliance and effectiveness.

From Passive to Proactive Governance

Initially, governance was reactive—triggered only when storage watermarks approached critical levels. The new approach predicts water‑level risks, issues early warnings, and defines tiered response mechanisms, turning reactive fixes into proactive planning.

Multi‑Dimensional Governance

To scale governance across storage, compute, and traffic, Bilibili unified tag models, quota controls, and SOPs. Tags map assets to indicators, enabling rapid strategy generation. Quota limits are applied to storage, offline/real‑time compute, and traffic, with hierarchical enforcement from company to department to workspace.

Tooling and Automation

The Governance Center aggregates problem discovery, strategy explanation, alerts, execution, and benefit reporting into a single UI, allowing users to receive tasks, act on them, and view outcomes without juggling multiple tools.

Future Directions

Upcoming focus areas include shifting from cost‑only to cost‑and‑value evaluation, integrating fragmented governance into a unified center, and adopting lake‑house and one‑service architectures to improve data reuse and ROI.