How We Built a Scalable, Low‑Latency Ranking System for Millions of Users

Background

Leaderboards satisfy users' competitive instincts and are required by many products. QQ membership, QQ Anime, Penguin E‑Sports, and game events all need ranking services, driving rapid growth from a few leaderboards to tens of thousands, with peak user counts reaching 90 million and active storage clusters handling billions of users.

How to support near‑by access with low latency?

How to schedule tens of thousands of leaderboard requests automatically?

How to reduce machine cost by choosing appropriate storage engines?

How to prevent resource contention between services?

The following sections detail how each challenge was addressed.

Basic Architecture of the Ranking System

The system consists of several services:

Access Service (stateless RPC interfaces for querying ranks, top‑N, etc.)

Storage Service (stateful, master‑slave deployment for leaderboard data)

APIServer (APIs for leaderboard creation, business onboarding, configuration, capacity)

Scheduler Service (selects optimal storage nodes based on business requirements)

Agent (reports storage node capacity and monitoring data)

ZooKeeper (stores routing and capacity data)

MySQL (stores business onboarding info, leaderboard configurations, monitoring metrics)

When a business integrates, it receives a business ID and a leaderboard ID, then calls the Access Service via L5. The service looks up routing information in ZooKeeper and interacts with the appropriate Storage Service to read or write leaderboard data.

How to Support Near‑by Access and Low Latency

Business services are deployed in Shenzhen and Shanghai, where intra‑city ping is ~30 ms, exceeding the desired <5 ms latency. Early deployments kept routing data only in Shenzhen, but as traffic grew, Shanghai experienced latency spikes when the local cache missed. The solution was regional autonomy: each region runs its own routing data and storage nodes, synchronized via ZooKeeper’s Zab algorithm across seven nodes in four IDC locations, achieving sub‑2 ms latency.

How to Support Tens of Thousands to Millions of Ranking Requests

Ranking request handling evolved through three stages:

Stone Age: manual configuration of a few leaderboards.

Bronze Age: dozens of leaderboards approved via a web UI.

Iron Age: tens of thousands of leaderboards automatically scheduled, with capacity planning and no manual intervention. Two request paths exist: a web portal for occasional requests and an API for high‑frequency, large‑scale requests. The scheduler evaluates business parameters (region, expected users, request rate, leaderboard type, container deployment, storage engine) and selects storage nodes through filtering (health check, tag matching, capacity check) and scoring (minimum‑resource, weighted‑mix algorithms). Selected nodes are recorded in ZooKeeper, and agents periodically report capacity.

How to Reduce Machine Cost and Choose the Right Storage Engine

Leaderboard operations include rank queries, score updates, top‑N retrieval, and deletions. Depending on usage patterns, different storage engines are appropriate:

Redis sorted sets (zset) provide O(log N) operations for both score updates and rank queries, ideal for latency‑sensitive workloads.

LevelDB/RocksDB store simple key‑value pairs; with encoding tricks (e.g., SSDB) they can emulate sorted sets but rank queries become O(N), suitable for workloads that rarely need ranking.

Cost reduction strategies:

Select storage engines per business need—use SSDB/LevelDB for leaderboards that do not require rank queries.

Implement hot‑cold data separation: hot data stays in Redis, while cold data (e.g., inactive event leaderboards) migrates to SSDB/LevelDB, freeing memory.

How to Prevent Resource Contention Between Services

To avoid one business monopolizing storage capacity, the system enforces quotas and isolation via Docker containers. Containers run in host network mode for performance, use AUFS for image storage, and are managed by a Daemon engine. Resource limits are applied via cgroups, and containers can be dynamically resized or added as demand grows.

Conclusion & Future Plans

Through iterative improvements, the ranking system now offers high availability, sub‑2 ms latency, low cost, automatic onboarding, scheduling, disaster recovery, resource isolation, monitoring, scaling, and hot‑cold data separation. Future work includes full automation of master‑slave failover (considering master_repl_offset lag), Ceph‑RBD backed volumes for containers, and online migration of entire leaderboards.

Source: CSDN, author: Tang Cong