Knowledge Distillation Techniques for Recommendation Systems: Methods, Scenarios, and Practical Insights

With the rapid development of deep learning, models such as ResNet for images and BERT for natural language processing have achieved impressive performance, but their increasing depth and parameter count cause latency problems when deployed at scale. Knowledge distillation offers a solution by training a lightweight student model under the guidance of a powerful teacher model.

The classic teacher‑student paradigm treats the teacher’s logits (pre‑softmax scores) as "dark knowledge" that the student tries to mimic. Hinton’s original work introduced temperature‑scaled softmax to soften the probability distribution, allowing the student to learn richer information from the teacher.

In recommendation systems, three cascading stages—recall, coarse‑ranking, and fine‑ranking—can benefit from distillation. Fine‑ranking, which demands high accuracy, often uses complex models; distilling these into a compact student reduces inference latency while preserving quality. Two typical scenarios are upgrading from non‑DNN to DNN models and improving an already simple DNN model.

For recall and coarse‑ranking, distillation can be performed either jointly (training teacher and student together with shared embeddings) or in two stages (first training a teacher, then using its logits or ranking lists to supervise the student). Joint training enables additional tricks like feature‑level distillation, while two‑stage training saves resources.

Distillation objectives can be categorized as point‑wise, pair‑wise, or list‑wise. Point‑wise treats the teacher’s top‑K items as positive examples and the rest as negatives; pair‑wise (e.g., BPR loss) constructs item pairs to preserve order; list‑wise (e.g., NDCG) attempts to match the entire ranked list, often requiring a mapping from logits to relevance scores.

Practical implementations from Alibaba (Rocket Launching) and iQIYI (dual‑DNN ranking) demonstrate that student models can achieve 5× faster inference and comparable accuracy. Real‑world experiments on platforms such as Weibo show 2‑6% improvements in click‑through and interaction metrics when applying point‑wise distillation to recall models.

Finally, a unified “one‑teacher‑three‑students” framework is proposed, where a powerful teacher jointly trains student models for recall, coarse‑ranking, and fine‑ranking, aligning objectives across the pipeline and maximizing both efficiency and performance.