How Bilibili Scaled Its Comment System with Multi‑Level Storage and Automatic Failover

Background

Comments are a core part of Bilibili’s ecosystem, driving user interaction, recommendation, and community culture. During hot events, comment read/write traffic spikes dramatically, stressing the backend. The service relies heavily on cache hits; when Redis misses, queries fall back to TiDB, whose failures can render the comment service unavailable.

Architecture Design

The original architecture uses Redis for sorted‑set indexes (likes, time, hotness) and TiDB for persistent storage. Large comment sections cause cache misses, leading to expensive TiDB queries that degrade performance and threaten throughput.

To eliminate TiDB’s single‑point‑of‑failure, a multi‑level storage system was built on Bilibili’s self‑developed Taishan KV store. The design goals are disaster‑recovery, automatic degradation, and better query performance under high load.

Storage Design

Two abstract data models are defined:

Index : stored as a Redis Sorted Set (e.g., like‑count ordering).

KV : stored as a key‑value pair in Taishan, containing comment metadata and content.

Queries first retrieve comment IDs from the Index, then fetch full comment data from KV. This separation allows each model to use the storage structure best suited for its access pattern.

Moving from SQL to NoSQL improves latency: Taishan’s Sorted Set P999 latency is ~10 ms, while KV lookups are ~5 ms, far better than TiDB’s multi‑second slow queries.

Data Consistency

Synchronizing TiDB (structured) and Taishan (unstructured) requires a custom pipeline because no off‑the‑shelf tool exists. Challenges include data loss, write failures, write conflicts, out‑of‑order processing, and sync delays.

To mitigate write failures, a retry queue is introduced. Concurrent writes can cause race conditions, addressed with a Compare‑and‑Swap (CAS) approach and version numbers.

UPDATE reply SET like_count=like_count+1, version=version+1 WHERE id = xxx

Example of out‑of‑order updates:

Binlog 0 (like_count = 1) fails, goes to retry queue.

Binlog 1 (like_count = 2) succeeds, setting count to 2.

Retry queue processes Binlog 0 later, overwriting count back to 1, causing inconsistency. Version checks ensure only newer updates are applied.

Reconciliation

Both real‑time and offline reconciliation are employed:

Real‑time: TiDB binlog events trigger a delayed queue; each event is compared with the corresponding Taishan record, and anomalies are flagged for repair.

Offline: Daily T+1 data warehouse comparisons validate final consistency.

Degradation Strategy

High availability demands automatic fallback from the primary store to the secondary when errors or timeouts occur. Two classic strategies—serial and parallel—were evaluated. Serial is simple but slow; parallel is fast but wastes resources. The chosen “hedging” policy sends a backup request to the secondary after a short delay. If the primary responds first, its result is used; otherwise the backup result is returned. This balances latency and resource usage. In production, TiDB is primary for latency‑sensitive, lightweight queries, while Taishan becomes primary for heavy analytical workloads. During a TiKV outage, the system automatically degraded to Taishan, keeping the comment service stable.

Summary & Outlook

The multi‑level storage architecture, combined with robust consistency mechanisms and a hedging degradation policy, dramatically improves the stability and performance of Bilibili’s comment service, ensuring users enjoy a responsive and reliable commenting experience.