Deep Learning for Big Data Recommendation Systems: Tencent's Industrial Practice

This article shares Tencent's practical experience in applying machine learning and deep learning technologies to recommendation systems in big data scenarios.

Understanding Recommendation: In industrial practice, recommendation is not merely a scoring problem. From a business perspective, recommendation can be viewed as a marketplace with multiple participants whose demands often conflict. The real challenge is achieving the "Best Match" - matching different stakeholders'诉求 (demands). To accomplish this, Tencent employs an Offline-Nearline-Online architecture, implemented through the "神盾推荐系统" (Shield Recommendation System), which handles hundreds of billions of daily requests across Tencent's ecosystem.

From Recommendation to Deep Recommendation: The "depth" in big data deep learning has two meanings: first, deeper semantic understanding of users, content, and context through mapping to semantic space; second, addressing fundamental challenges in big data including complex data forms, personalized fine-grained learning, and dynamic data distribution shifts (concept drift).

Three AI advantages are applied: (1) Perception - embedding dissimilar concepts into similar semantic spaces, such as mapping Trump and Obama close together as both are US presidents; (2) Learning - using large-scale neural networks with high VC dimension to find optimal functions, with constraints and network design to control variance; (3) Decision - applying reinforcement learning for dynamic strategy selection, which is particularly valuable in scenarios like Weishi recommendations that emphasize exploration and diversity.

DeepR Framework: Tencent developed DeepR, a deep learning framework specifically designed for big data scenarios. Its core component is RCaffe, a customized deep learning framework based on Caffe. Key mechanisms include: grouped data input for flexible processing; BDM (Batch Dynamic Model) mechanism for handling large-scale model parameters by loading only required parameters for current batch; hybrid data and model parallelism for multi-GPU communication; and optimized storage for large-scale sparse data.

Key Takeaways: In big data deep learning, "depth" means deeper semantic understanding of users and content, not just model depth. Data determines the upper bound of效果 (performance); models only fit the data. Engineering capability determines the practical application level of AI.