Bilibili's Big Data Development Governance Platform: Architecture, Challenges, and Strategies

Platform Overview – The Big Data Development Governance Platform built by Bilibili provides a one‑stop solution for massive data transmission, storage, query, development, testing, publishing, management, and operations, serving various internal roles that require data.

Four Main Parts – (1) Introduction of the platform, (2) Pain points in platform construction, (3) Data‑governance‑driven methodology, and (4) Product promotion strategy.

Key Components – The platform consists of two major modules (development and governance), three scenarios (data production, consumption, and governance), and four user groups (data analysts, data RDs, algorithm RDs, and data operations). Development focuses on IDE, job scheduling, and code management, while governance covers data management, security, cost, and quality tools.

Construction Pain Points – Company‑level challenges include platform stability, rapid business iteration, data‑driven maturity, and organizational complexity. Specific issues are insufficient stability, reliance on self‑built clusters, limited integration needs, and low data‑governance cooperation.

Data‑Driven Methodology – Organize a centralized platform team, adopt four methodological pillars (data model standards, data quality, cost, and security), and build supporting tools for data integration, development, operations, and governance.

Platform Scale and Evolution – Bilibili operates over 10,000 physical nodes, processes >1 trillion new records daily, and runs >150,000 pipelines. Evolution milestones span from 2017 general enablement, 2019‑2020 data quality focus, to 2021‑2022 cost‑driven governance.

Core Capabilities – Data integration (batch & streaming, CDC), job scheduling with time, dependency, and hybrid triggers, quality SLA monitoring (MTTD, MTTR), intelligent baselines, DQC, and automated back‑fill tools.

Cost Governance – Asset ownership clarification, usage‑based billing (CPU, memory, storage), and workspace isolation to allocate resources per department, enabling transparent cost reporting and optimization.

Product Strategy Insights – Emphasize business collaboration, cautious adoption of new technologies, organizational alignment, prioritization of quality & cost over security, and the importance of data‑driven operations rather than pure tooling.

Q&A Highlights – Discussed tool promotion, open‑source component recommendations (Flume, Flink, Kafka, DataX, Waterdrop, Airflow, DolphinScheduler, ClickHouse, Iceberg, Hudi), and multi‑tenant resource isolation strategies using queue‑based prioritization (P0, P1, P2).