Optimizing Bilibili Video Detail Page Backend with Business Association Index and Data Aggregation Gateway

This article analyzes the implementation technology of Bilibili's most frequently visited video detail page and the problems caused by the continuous growth of fan‑out reads. It proposes a business‑association index that reduces service load by more than 90% and presents a generic Data Aggregation Gateway (DAGW) solution for broader aggregation scenarios.

Business Background Bilibili is a PUGV‑centric video community. The video detail page is the main user interaction point. As the platform expands, many additional features (topics, video honors, notes, user avatars, etc.) are attached to the detail page, dramatically increasing traffic and complexity.

Current Architecture The user‑facing architecture is divided into four layers: Terminal Layer: mobile apps, H5, PC web, TV, car‑display, speakers, etc. Access Gateway: LB + API‑Gateway handling routing, protocol conversion, rate limiting, security, etc. BFF (Backend for Frontend): Separate BFF services per terminal (web‑interface, app‑interface, tv‑interface) and per functional area (app‑feed, app‑view, etc.). Business Service Layer: Domain‑specific services providing business capabilities.

Problems Identified 1. Fan‑out read explosion : As more downstream services are called from the video detail BFF, the read fan‑out grows, causing massive traffic load and increased system complexity. 2. Latency degradation : The increasing number of fan‑out calls leads to higher latency and time‑out probability, eventually becoming unacceptable for users.

Analysis & Modeling Many downstream services return empty responses for videos that are not associated with them. By profiling response sizes with Prometheus histograms, it was found that over 90% of service responses are empty. This indicates that the BFF does not know which services are actually relevant for a given video.

To address this, a sparse vector called a video‑business index is introduced, mapping each video ID to the set of associated business services.

Implementation The index is stored in Redis using a hash map. Example command:

HMSET index_vid1234 biz1 0 biz2 1 bizM "hot"

In addition to Redis, an internal KV store is used for persistence and Redis fail‑over. The index is built from full‑batch and incremental imports of downstream service association data.

Architecture Extension A generic Data Aggregation Gateway (DAGW) is designed to serve aggregation needs across multiple detail pages (video, short video, live, user profile) and terminals. Requirements include: Terminal‑specific on‑demand aggregation models. Low‑cost addition of new business scenarios. Integration with the business‑association index for load reduction.

Compared solutions such as GraphQL, protobuf field masks, and View Enum are discussed. The article adopts a View Enum approach; the enum definition is shown below:

enum ArchiveView { // Unspecified, no data returned UNSPECIFIED = 0; // Common view definitions // Simple archive info for queries SIMPLE = 1; // Basic info for list pages BASIC = 2; // Basic + part info for sharing BASIC_WITH_PAGES = 3; // Full detail for APP ALL_APP = 4; // Full detail for WEB ALL_WEB = 5; // Full detail for TV ALL_TV = 6; // Future extensions }

Effects After deploying the DAGW and the business‑association index, 30+ downstream services have been integrated, achieving over 90% traffic and load reduction on average. Specific results include: 1. Video “high‑energy highlights” service: peak QPS >100k reduced by 99%. 2. Fan badge service: traffic reduced by 85%.

References 1. The Tail at Scale – https://research.google/pubs/pub40801/ 2. GraphQL: From Excitement to Deception – https://betterprogramming.pub/graphql-from-excitement-to-deception-f81f7c95b7cf 3. View Enum – https://google.aip.dev/157