Bilibili Data Quality Assurance System: Architecture, Practices, and Case Study

Introduction: Bilibili's data quality assurance system aims to ensure reliable data across its data warehouse and modeling processes.

Background and goals: The system evolved through database, data warehouse, data platform, and middle‑platform stages, each with increasing data volume and quality requirements.

Architecture: The platform consists of four layers—data sources, data platform, middle platform, and data applications—integrating services such as account, tracking, CRM, and third‑party systems.

Core capabilities: (1) a quality data warehouse that centralizes monitoring data, baseline services, DQC, lineage, and incident management; (2) a comprehensive monitoring and rule engine covering completeness, consistency, validity, timeliness, and cross‑component checks; (3) cross‑team collaborative safeguards and standardized SLA mechanisms.

Digital‑driven continuous optimization: Metrics are defined, measured, and visualized through quality scores, enabling automated detection, root‑cause analysis, and iterative improvement of data pipelines.

Efficient incident handling: Night‑shift procedures include rapid alert response, impact assessment, coordinated recovery, and post‑mortem documentation, reducing night‑shift frequency and incident resolution time.

Case study: A real‑world example shows challenges such as low monitoring coverage, fragmented alert handling, and high night‑shift load, and how the introduced framework improved coverage, reduced incidents by 50 % and cut night‑shift time by 86 %.

Future outlook: Plans include expanding coverage, enriching safeguard strategies, advancing tool‑based automation, and moving from manual to information‑driven and eventually intelligent quality assurance.

Q&A: Addresses alignment of table‑level quality rules, cross‑platform real‑time task evaluation, and tools for accelerating root‑cause analysis.