How Alibaba Youku Guarantees Real‑Time Data Quality for Massive Video Search

Background

During the pandemic, online video consumption exploded, creating a massive demand for accurate, timely search results on Youku. The platform processes billions of video items and an even larger amount of metadata, making real‑time data quality a critical challenge.

Current Situation

The Youku video search pipeline involves dozens of intermediate tables and a streaming architecture that decouples layers for high real‑time performance, but this complexity introduces significant quality‑assurance difficulties.

Real‑Time Data Quality Assurance Scheme

The quality goals are threefold:

Basic data content quality

Correctness and timeliness of streaming links

Non‑negative impact of data changes on business outcomes

An online‑offline‑full‑link closed‑loop framework was designed to meet these goals.

Offline Quality Checks

1. Real‑time Dump

Testing covers link‑node comparison, timeliness, correctness, consistency, and availability, using a real‑time dump solution.

2. Data Consistency

Ensures each node consumes data consistently by comparing consumption across time and frequency.

3. Data Correctness

Prioritize data that directly affects user experience.

Guarantee core business data integrity.

Apply generic and business‑specific rules to middle‑layer data and perform diff checks.

4. Data Availability

Efficient read/write storage.

Consistent service interfaces (API, PB, SDK).

Secure and reliable storage to prevent unauthorized modifications.

5. Timeliness

A trace‑plus‑wrapper model captures the processing time of each node, outputting JSON‑formatted trace information for easy analysis.

Performance Testing

The full‑link service consists of the Bigku reverse‑lookup (HSF) service and Blink computation nodes.

Two data‑generation approaches are used: synthetic message simulation and replay of real dump data. Load can be generated via a dedicated service interface or Blink’s message replay mechanism.

Online Quality Assurance

Service Stability

Monitors both streaming task node stability and internal service health.

Entity Services

HSF services are observed via Alibaba’s unified monitoring platform.

Data Consumption Guarantee

Core layer tracks message volume and categories; middle layer records accept, success, fail, and skip metrics.

Data Content Guarantee

Three components work together:

Sampler – extracts IDs from real‑time Blink streams using interval or random strategies.

Data‑monitor – checks update timeliness and feature attributes.

Effect‑monitor – evaluates recall, ranking, and user‑experience impact.

Real‑Time Intervention & Auto‑Repair

Provides three intervention modes: normal ID update via main link, precise content‑level intervention, and emergency VIP channel for forced updates when the main path is blocked.

Quality Efficiency

Real‑Time Debug

Based on the real‑time message channel, developers can enable debug mode to view detailed processing steps without additional integration cost.

Full‑Link Trace

A generic trace model captures the entire data flow across four business blocks, providing JSON‑formatted trace info for each node.

Examples of service‑level data include bigku_service (message mirror), mid_show_f (basic features), and sum_video_f / ogc (search‑link data).

Afterword

Data is the lifeblood of algorithms; ensuring its quality improves content distribution, retains users, and delivers high‑quality video experiences even during special periods like a pandemic. Future work will deepen node‑level quality checks and explore the relationship between massive data and user perception.