HRL-Rec: A Hierarchical Reinforcement Learning Framework for Integrated Recommendation

Real‑world information‑flow recommendation systems must simultaneously recommend heterogeneous items (articles, videos, news, products, etc.) from multiple sources, often using separate channels to decouple and customize models. In such integrated recommendation scenarios, both item‑level and channel‑level user preferences are crucial.

To address these challenges, we propose HRL-Rec, a hierarchical reinforcement learning framework consisting of a low‑level channel selector (LRA) and a high‑level item recommender (HRA). The channel selector generates a sequence of channels for each request, while the item recommender selects specific items constrained by the chosen channels. Diverse loss functions, supervised and similarity losses, and multiple reward signals (clicks, dwell time, diversity, novelty) are designed to ensure fast, stable convergence.

HRL-Rec is built on a list‑wise recommendation pipeline. For position t in the list, the channel selector predicts channel c_t based on previously recommended items and user/context features; the item recommender then selects item d_t within channel c_t . Both agents are trained with DDPG, augmented by the additional losses to improve realism of generated actions.

The overall architecture (Figure 2) shows how HRL-Rec processes each user request, extracts features from historical items, encodes them with a sequential model, and feeds the state to actor‑critic networks for both channel selection and item recommendation.

Extensive offline and online experiments on the WeChat "See" system demonstrate that HRL-Rec achieves the best performance across all metrics, including click‑through rate, average clicks, and diversity. Ablation studies confirm the effectiveness of each module, and stability tests show that HRL‑Rec maintains consistent channel proportions over two weeks, indicating strong system robustness.

In summary, HRL‑Rec systematically tackles the integrated recommendation problem by modeling multi‑granular user preferences with hierarchical reinforcement learning, delivering superior accuracy, diversity, and stability, and has already been deployed to serve millions of users.