Designing a 200M‑Request Recommendation System with 50ms P99 Latency

Introduction

Last year the team took over a content platform with 80 million daily active users. The legacy system could not handle peak traffic: the recommendation API P99 latency spiked to 800 ms and cache breakdown caused two outages per day. After more than half a year of redesign, daily request volume grew from 30 million to 200 million and P99 latency stayed under 50 ms.

1. What Does Hundred‑Million‑Level Traffic Mean

Two hundred million requests evenly spread over 24 hours yield about 2,300 QPS, but actual traffic is highly skewed. The evening peak (20:00‑23:00) reaches four to five times the average, easily exceeding 10,000 QPS. Each request may trigger 3‑5 model inferences, dozens of feature lookups, and several cache reads, so internal call volume is more than ten times the external request count.

The three dimensions that “hundred‑million‑level” traffic stresses are throughput, latency, and stability, which trade off against each other.

2. Overall Architecture Design

The final architecture consists of four layers: ingress, recall, ranking, and data. Requests flow from the client through a gateway to the recall layer, then through coarse and fine ranking, and finally a mixed‑ranking output. The end‑to‑end latency is kept within 50 ms.

3. Recall Layer: Quickly Filtering Tens of Millions of Items

The recall problem is to filter a few thousand candidates from tens of millions (or even a billion) items within 10 ms. A parallel multi‑path recall strategy is used, where each path runs independently and results are merged and deduplicated.

Vector Recall is the core channel. Instead of the older Faiss solution, Milvus 2.5 with DiskANN index is employed, supporting billion‑scale vectors with millisecond‑level search. Switching the embedding model from a dual‑tower DSSM to a fine‑tuned pre‑trained language model improves recall accuracy by roughly 15 %.

Real‑Time Interest Recall updates a short‑term interest vector for each user in real time. Apache Flink consumes user‑behavior streams, updates the vector, and writes it to Redis, enabling the latest interests to be used instantly during recall.

Graph Recall leverages NebulaGraph to build a heterogeneous “user‑item‑tag” graph. A Node2Vec‑style walk extracts high‑order associations, adding diversity and surfacing items the user does not explicitly know they might like.

After merging the multi‑path results, typically 3,000‑5,000 candidates are passed to the ranking layer.

4. Ranking Layer: Engineering the Fine‑Grained Model

Ranking is split into coarse and fine stages. Coarse ranking uses a lightweight model (e.g., a dual‑tower with simple cross features) to shrink candidates from a few thousand to a few hundred within 5 ms.

Fine ranking adopts the latest Transformer‑based sequence modeling. The team uses a customized Behavior Sequence Transformer (BST) that ingests the most recent 200 user actions, delivering a 3‑4 percentage‑point AUC lift over classic DeepFM/DCN structures.

Model inference is optimized by compiling PyTorch models with torch.compile, deploying them on NVIDIA Triton Inference Server, and applying TensorRT‑LLM for operator fusion and FP8 quantization. A single A100 GPU handles up to 8,000 inference requests per second. Model updates combine hourly incremental training with daily full‑batch training to capture short‑term trends without over‑fitting.

After fine ranking, a rule‑based mixing stage shuffles similar content, inserts operational slots, and controls ad ratios, ensuring a varied user experience.

5. Online Service: Key Techniques for Low Latency and High Availability

Keeping the whole pipeline under 50 ms requires more than model optimization.

Multi‑Level Caching forms the performance backbone: an in‑process Caffeine cache (≈40 % hit rate) stores hot item features, a Redis cluster (≈95 % hit rate) holds user profiles and item attributes, and only a fallback to the feature store incurs remote access. This reduces average feature query latency from 15 ms to 2 ms.

Timeout Degradation assigns independent 8 ms timeouts to each recall path; timed‑out paths are dropped without affecting others. If the fine‑ranking model fails, a pre‑computed “hot + personalized” list is returned, preferring availability over perfect relevance.

Predictive Autoscaling uses Kubernetes HPA for reactive scaling, but because pod startup plus model loading takes ~30 s, a proactive scaler predicts traffic spikes from historical patterns and adds resources 15 minutes in advance, eliminating scaling‑induced latency spikes during peak hours.

6. Feature Engineering and Real‑Time Data Streams

Features are categorized into three freshness tiers:

Real‑time features (second‑level updates): last 5 minutes of clicks, current session statistics, aggregated by Flink from Kafka and written to Redis.

Near‑line features (minute‑level updates): 24‑hour interest distribution, hourly item click‑through rates, computed by Flink and stored in HBase.

Offline features (day‑level updates): long‑term user profiles, item categories, author reputation, refreshed nightly by Spark.

Since 2025 the team has used Feast as a unified Feature Store, eliminating training‑serving skew. Feature registration, versioning, and consistency checks are all managed within Feast.

7. System Observability and Disaster Recovery

Observability relies on three pillars: Metrics via Prometheus + Grafana, Tracing via OpenTelemetry + Jaeger (each request traces which recall paths were taken and per‑step latency), and Logging via ELK. Multi‑level alerts trigger group messages at P99 > 80 ms and phone alerts at P99 > 150 ms.

Disaster‑recovery lessons:

Fast model rollback: after a fine‑ranking model caused a 2 % CTR drop, the system now automatically performs a 5 % gray rollout and rolls back if core metrics regress, without human intervention.

Cache avalanche protection: popular item cache TTL is set to 30 minutes ± 5 minutes random jitter to avoid simultaneous expirations.

Active‑active deployment: full recommendation services run in two regions with DNS‑based traffic steering; a data‑center failure triggers a 30‑second cut‑over with no user impact.

8. Conclusion

The most expensive resource turned out to be debugging time rather than hardware. Key takeaways are: avoid chasing a perfect architecture from day one; build AB‑testing infrastructure early; and monitor negative recommendation effects such as content monotony and unwanted items, as diversity and feedback handling impact retention more than raw CTR.