Evolution of Real‑Time Game Recommendation System at NetEase Leihuo

China's massive game user base provides abundant data, and with rapid internet growth the industry has advanced from traditional industrialization to leveraging big data, data mining, and recommendation algorithms for the next wave of development.

Before online inference, a classic offline recommendation workflow is introduced: user behavior logs are collected via message middleware, ETL‑processed into a data warehouse, transformed into feature stores, used to update models, and batch‑predicted results are stored in a business database (e.g., Redis) for client queries.

Offline recommendation suffers from two main issues: the T+1 feature update introduces latency, and static features may not reflect the latest user state, leading to mismatches between recent behavior and predictions.

The proposed solution adds real‑time feature engineering to the recommendation system, capturing users' current actions to provide more accurate recommendations and largely solving the latency and relevance problems.

The online inference service, the core of the real‑time system, follows a pipeline designed for game scenarios: recall generates candidate items, a ranking module scores them, and a re‑ranking module adjusts scores based on business rules before returning results to the client.

Practical experience includes several optimization techniques: batch prediction performance is improved by reducing feature I/O, using pipelines, and multithreaded asynchronous prediction; frequently unchanged features are cached to avoid repeated database hits; and single‑model inference time is reduced through model compression, feature reduction, and thread‑pool tuning.

Business‑level insights cover transitioning from single‑model to multi‑model loading to support diverse recommendation scenarios, managing multiple engineers’ models via branch development, and operational tricks such as warm‑up calls to speed up initial responses and externalizing score coefficients to a database to avoid frequent deployments.

Future optimization directions focus on incorporating real‑time training and sample generation: as real‑time ETL produces features, it also creates samples for immediate model updates, enabling the system to quickly adapt to new player behaviors and mitigate model degradation.

Follow the "NetEase Leihuo UX User Experience Center" for more technical articles and insights.