Inside Industrial-Scale Recommendation Systems: From Millions of Items to a Few Real-Time Results

1. Recommendation systems are engineering systems first

Many teams blame stagnant online metrics on outdated algorithms, but the author argues that a robust engineering foundation and a complete data loop are prerequisites for any algorithmic improvement. An industrial‑grade system couples online services, data computation, content governance, user modeling, real‑time feedback, and distributed engineering into a single closed‑loop.

2. The ultimate goal: real‑time dynamic matching of users and content

The core mission is to build an efficient, iterative bridge between massive users and massive items, answering three questions: who to recommend, what to recommend, and how to rank.

2.1 Who to recommend – comprehensive user understanding

Long‑term interests : derived from historical clicks, stays, collections, follows, etc., representing stable preferences.

Short‑term interests : captured from the current session’s clicks, rapid swipes, and temporary browsing, reflecting momentary intent.

User profile composition : includes basic attributes, behavior tags, preference tags, sequential vectors, and embeddings.

2.2 What to recommend – standardized content understanding

Basic content features : category, keywords, entities, publish time, popularity, region, etc., for coarse filtering.

Content quality features : authority, originality, completeness, sentiment, freshness, compliance, and a composite quality score, used to filter low‑quality or harmful items.

Content standardization process : unified ingestion, parsing, cleaning, deduplication, tagging, and semantic vector encoding, stored in content and vector stores for downstream recall and ranking.

High‑quality, structured content directly determines the ceiling of recommendation performance.

2.3 How to rank – multi‑objective decision making

Focusing solely on click‑through rate leads to content homogenization and user‑experience degradation. Industrial ranking balances user experience (CTR, dwell time, completion, interaction, diversity, freshness) with platform business goals (cold‑start support, vertical ecosystem, compliance, fair traffic distribution).

3. Four‑layer industrial architecture

The architecture follows a decoupled, responsibility‑single, offline‑online collaborative design.

3.1 Content production & governance layer (data source)

Handles multi‑channel content ingestion, automated parsing, quality inspection, structured tagging, cleaning, deduplication, indexing, and unified storage, turning raw, unstructured material into machine‑readable assets.

3.2 Offline data & feature layer (system brain)

Performs full‑scale data cleaning, positive/negative feedback aggregation, batch feature generation, long‑term user portrait updates, interest weight calculation, offline candidate generation, and model training. It combines batch processing with near‑real‑time computation to keep both accuracy and timeliness.

3.3 Online recommendation service layer (decision core)

Meets high concurrency, low latency, high availability, and stability requirements. A single request follows the chain: gateway routing → user profile loading → parallel recall (long‑term, hot, collaborative, exploratory) → hierarchical ranking (coarse, fine, re‑rank) → rule‑based tweaks → cache write → response.

3.4 Real‑time feedback loop layer (evolution engine)

Every user action (click, stay, read, like, follow, skip, long‑press, exit, report) is asynchronously collected, stored, and fed back to update features, user portraits, model parameters, and candidate sets, forming a "request‑recommend‑feedback‑optimize‑re‑recommend" loop.

3.5 Real‑world case

DAU: 30 million+

Peak QPS: 30 000+

Content pool: millions of items

P95 latency: <300 ms

The system’s layered design, data flow, and service scheme match the architecture described above.

4. Core principle: offline learning, online decision

Offline components handle heavy, full‑data jobs (log processing, feature production, long‑term profiling, model training, batch candidate generation). Online components handle millisecond‑level, high‑throughput inference (real‑time state loading, parallel recall, multi‑stage ranking, rule tweaks, response).

Data flows from real‑time user behavior → log collection → message queue → offline batch/near‑real‑time processing → feature/model updates → online service loading → next recommendation.

5. End‑to‑end request lifecycle

When a user opens the app and pulls to refresh, the system completes the following steps within tens of milliseconds:

Request entry & traffic control : authentication, rate limiting, degradation, routing.

User state fast loading : parallel fetch of long‑term profile, behavior sequence, current session from high‑performance KV store.

Parallel multi‑route recall : match long‑term interests, hot content, similar users, real‑time scenarios to produce hundreds of candidates.

Three‑stage ranking fusion : coarse filter, deep model scoring, business‑rule re‑ranking for diversity and freshness.

Result optimization & return : deduplication, expiration filtering, diversity calibration, cache write, response.

Real‑time feedback formation : capture subsequent user actions, push to feedback queue, drive feature/model updates for the next cycle.

6. Core engineering challenges

High concurrency & low latency : support tens of thousands of QPS with P95 latency <100 ms, requiring extreme optimization of recall, KV queries, and model inference.

Feature consistency : offline training features must exactly match online inference features; any drift causes severe metric drops.

Real‑time user interest : capture interest shifts within minutes or seconds to adjust recommendations promptly.

Cold‑start adaptation : generate reasonable recommendations for new users or new items lacking interaction data.

Iterability & explainability : provide traceable, explainable results and a complete A/B testing framework for safe, rapid version upgrades.

7. Summary

The essence of an industrial‑grade recommendation system is an engineering foundation that enables algorithmic capability, driven by a closed data loop. Stable layered architecture, smooth data flow, robust engineering base, and continuous feedback are the pillars that allow long‑term iteration and sustained business value.

To truly master recommendation systems, one must look beyond isolated model tweaks and adopt a holistic architectural perspective, understanding layer responsibilities, offline‑online collaboration, and data‑loop mechanics.