Design and Optimization of Bilibili's Real-Time Data Quality Monitoring Platform

Background – Data quality is a prerequisite for reliable big‑data applications at Bilibili; the platform must provide real‑time, accurate data trusted by all business units.

Quality Platform Components – The DQC consists of data collection, inspection, and alerting, similar to traditional monitoring systems but tailored for big‑data workloads.

First‑Version Real‑Time DQC – Implemented with Flink, each rule created a separate Flink job writing results to MySQL, leading to low resource utilization, high network bandwidth consumption, and instability due to frequent restarts.

Design Goals for the New Solution

Start once without restarts to avoid resource contention.

Allow one task to serve multiple low‑traffic topics.

Perform multiple rule checks per consumption to reduce duplicate reads.

New Architecture – Introduces three topic categories (large, medium, small) and separates handling:

Small/medium topics: data is fully ingested into an InfluxDB full‑table, then aggregated via Continuous Query (CQ) into a CQ table.

Large topics: processed directly in Flink, aggregating results into the CQ table.

Dynamic topic and rule management is achieved through configuration‑center updates without restarting Flink jobs.

InfluxDB Proxy – A proxy service compresses and batches writes, performs dual‑write for consistency, retries failures, and selects optimal backend nodes for reads. Optimizations reduced request size and eliminated redundant tags, cutting network I/O by over 90%.

Data Model

Full‑table stores time, subtask, sinknum, tags, record_num, and field values.

Tags are chosen for low cardinality (e.g., platform), while high‑cardinality fields (e.g., mid) remain as fields.

CQ table holds time, rule_id, and aggregated value with a 14‑day TTL.

Optimizations for Data Inflation – Flattened output format changed to <groupKey, RuleIdList, Data> and a map‑reduce aggregation pipeline was added, significantly lowering the data‑inflation factor.

Operational Safeguards – Monitoring includes Flink back‑pressure, InfluxDB cluster health, sequence count, and handling of traffic spikes, crashes, and duplicate consumption. Automatic alerts trigger resource reallocation for high‑priority (P0) topics.

Future Work – Focus on further engineering automation, refined task‑topic placement, horizontal scaling of InfluxDB, and minimizing impact of quality monitoring on normal production workloads.