Comprehensive Overview of Recommendation System Technologies and Their Evolution

The article introduces the overall development of recommendation system technology, referencing industry papers and external articles.

1. Recommendation System Architecture – The system consists of a data layer that collects real‑time user, item, scene, and device information, applies feature engineering (normalization, discretization, etc.), and stores features in KV stores like Redis or iGraph. The model layer trains offline models (TensorFlow, PyTorch, Alibaba PAI) and serves them online, supporting full‑link interventions across recall, coarse ranking, fine ranking, and re‑ranking, with additional strategies for diversity, trust, fatigue control, and pagination.

2. User Understanding – User understanding is divided into three layers: data, insight, and comprehension. The data layer gathers comprehensive, real‑time features; the insight layer extracts value, behavior, and lifecycle insights; the comprehension layer abstracts higher‑level understanding. Both explicit (behavior filtering, tag extraction, clustering, intent inference) and implicit (vector representations of multi‑modal behaviors) approaches are discussed.

3. Recall Techniques – Recall combines engineering (multi‑channel pipelines, vector retrieval) and algorithms. Four major algorithmic categories are described: traditional recall (popularity, collaborative filtering, content‑based), knowledge‑based recall (graph or rule‑based, offering strong interpretability), representation‑based recall (dual‑tower, graph‑based models like DeepWalk, Node2Vec, TransE, KGAT), and matching recall (NCF, TDM, DR) that require precise negative sampling.

4. Ranking Techniques – Ranking progresses from coarse to fine stages. Coarse ranking evolved from rule‑based to linear models (LR, XFTRL) to dual‑tower and FSCD models, and finally to high‑performance models such as COLD and AutoFAS, emphasizing feature crossing, performance, and sample bias mitigation. Fine ranking transitioned from linear models (LR, FM, GBDT, XGBoost) to deep models (MLP, Wide & Deep, DCN, DNN variants) and incorporates attention, graph neural networks, and multi‑task/multi‑objective learning (MMOE, PLE, MMGCN, LOGO).

5. Other Recommendation Algorithm Directions – Discusses cold‑start solutions (posterior statistics, attribute utilization, few‑shot heuristics), bias modeling (position, exposure, popularity bias) with models like click models and causal inference, and explainability (user‑item, text/visual, social explanations) to satisfy regulatory and user trust requirements.

6. Evaluation – Describes the evolution of metrics from overall indicators (CTR, CVR, ECPM, DAU, diversity, novelty, retention) to stage‑specific metrics for recall (Recall, Precision, F1, Hit Rate), coarse ranking (AUC, GUC, MAP, consistency with fine ranking), and fine/re‑ranking (AUC/GUC per segment, multi‑scenario evaluation). Emphasizes the combination of online A/B testing and offline metrics for comprehensive assessment.

The article concludes with gratitude to the audience.