QQ Music Recommendation Architecture: Challenges, Solutions, and Future Directions

Over the past year QQ Music launched dozens of new recommendation features such as audiobooks, AI playlists, podcasts, and community suggestions, which caused a sharp increase in recommendation requests and data volume, putting pressure on the existing recommendation services.

The team identified three main challenges: numerous entry points leading to duplicated development, fragmented traffic across many apps reducing ROI of algorithm optimization, and a wide variety of item types (songs, videos, live streams, posts) and recommendation formats (feeds, horizontal modules, radio, immersive video) that required a more efficient and reusable architecture.

To address these issues, the team built a machine‑learning platform called Cube Studio, an open‑source project based on Kubeflow and extended with cloud‑native components. The platform provides low‑code model development, AutoML hyper‑parameter tuning, distributed training, visual modeling, model deployment, hot‑loading, multi‑version management, and a rich library of classic recall and ranking models, all isolated and elastically scaled via Kubernetes and vGPU.

The data service layer consists of user‑profile, asset, feature, and metadata platforms, powered by Spark, Kafka, MongoDB, Redis, and other components. A classic Lambda architecture produces both batch views (processed with Spark/Hive) and real‑time views (processed with Flink/Spark Streaming). Data is stored in MongoDB, Elasticsearch, HBase, and Redis after careful trade‑offs among volume, latency, and cost. Performance optimizations include second‑level aggregation windows, protobuf + Gzip compression, and operator chaining to reduce serialization overhead.

For recommendation recall, two categories of algorithms are used: content‑based (profiles, tags, attributes) and model‑based (collaborative filtering, vector representations, graph methods). The recall framework was first built with simple template paths, then refactored into a directed‑acyclic‑graph (DAG) of operator nodes, allowing business teams to configure new recall paths by assembling plugins without writing code. The DAG engine handles candidate generation, filtering, coarse ranking, truncation, and merging, while integrating with offline model training, label retrieval, vector search, monitoring, circuit‑breaking, and fallback pools.

These architectural upgrades have enabled rapid rollout of new features such as MOO App, AI playlists, and genre recommendations, while reducing development effort and improving system stability. Future work will focus on adopting state‑of‑the‑art algorithms to boost recommendation quality and strengthening collaboration between the recommendation team and product owners.