Bilibili Data Quality Assurance System: Architecture, Practices, and Case Study
This article presents Bilibili's data quality assurance system, detailing its evolution across four data platform stages, the multi‑layer architecture, core capabilities such as a quality data warehouse, digital‑driven continuous optimization, and efficient incident handling, and concludes with a real‑world case study and future outlook.
Background and Goals
The article introduces the background and objectives of Bilibili's data quality assurance, outlining four historical stages of data infrastructure—database, data warehouse, data platform, and middle‑platform—each with increasing complexity and quality requirements.
System Architecture
The architecture consists of four layers: data sources, data platform, data middle‑platform, and data applications. Sources include account, tracking, CRM, and third‑party systems, feeding both offline and real‑time pipelines into the warehouse.
Quality Data Warehouse Construction
A dedicated quality data warehouse aggregates monitoring, baseline, DQC, lineage, and alarm services. It builds three layers (detail, summary, high‑level) and provides dashboards for quality metrics, real‑time monitoring, and alarm attribution.
Core Capabilities
Complete quality assurance system: rule libraries, monitoring, and incident attribution.
Digital‑driven continuous optimization: metric definition, analysis, problem discovery, solution implementation, and impact measurement.
Efficient incident handling: rapid response, damage assessment, notification, hand‑off, recovery, and post‑mortem.
Case Study
The case study describes challenges such as low monitoring coverage, incomplete SOPs for on‑call, high night‑shift rates, and frequent alarms. It details the steps taken to classify alarms, improve rule precision, and reduce incident volume, achieving over 50% reduction in incidents and night‑shift workload.
Future Outlook
Future work focuses on expanding coverage, enriching assurance strategies, advancing tool‑based automation, and moving from manual to information‑driven and eventually intelligent quality assurance.
Q&A
Answers address how Bilibili aligns quality rules across tables, integrates cross‑platform real‑time tasks into the quality warehouse, and accelerates root‑cause analysis through an incident knowledge base.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
DataFunTalk
Dedicated to sharing and discussing big data and AI technology applications, aiming to empower a million data scientists. Regularly hosts live tech talks and curates articles on big data, recommendation/search algorithms, advertising algorithms, NLP, intelligent risk control, autonomous driving, and machine learning/deep learning.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.