Reinforcement Learning in Recommendation Systems: Practice, Challenges, and Industry Advances

Background and Challenges – Modern recommendation systems must handle multiple tasks such as click, like, and gift predictions, often using multi‑task models and fusion ranking to combine predictions. Traditional static data methods struggle with reinforcement learning because the agent needs continuous interaction with the environment, and data collection is costly and noisy.

Exploration and Practice – The authors model the fusion ranking problem as an MDP, using an actor‑critic framework with offline training. Initial attempts with DDPG suffered from value‑function overestimation, leading to divergence; TD3 mitigated but not fully. Analysis revealed mismatches between sampled actions and the dataset, causing critic over‑confidence.

Offline RL Techniques – Three data pipelines are discussed: Final Buffer (exploratory data), Concurrent (limited exploration), and Imitation (expert data only). Offline RL often exhibits high value estimates despite poor performance, a phenomenon termed inference error, arising from distributional gaps between the dataset and the true MDP.

Mitigation Strategies – Two approaches are proposed: (1) collect near‑infinite data to match true transition probabilities (impractical), and (2) constrain the policy to actions frequently seen in the batch, using techniques like batch‑constrained Q‑learning, conditional VAE‑generated actions, and bounded bias adjustments.

Model Architecture – The system uses a CVAE combined with an actor to generate candidate actions; the critic selects the action with the highest Q‑value. Dual critics (TD3 style) reduce overestimation, and soft updates keep target networks stable without online interaction.

Training Process – Experiments compare BCQ, BCQ‑EE, CQL, and UWAC on random vs. expert data. BCQ shows better convergence than unconstrained TD3, while CQL offers higher online gains but less stability. Offline policy evaluation (OPE) is used to assess model performance before deployment.

Frontier Progress – Emerging directions include multi‑agent RL for joint recommendation and advertising, hierarchical RL for long‑term and short‑term user interests, and inverse RL to infer reward functions from user behavior. Recent works also explore preference‑based RL and better offline evaluation protocols.

Evaluation of RL in Recommendation – Traditional one‑step accuracy metrics are insufficient; the authors advocate offline policy evaluation, simulator‑based testing, and careful metric selection to capture long‑term user satisfaction.

Future Outlook – The authors highlight the promise of preference‑based and inverse RL, the need for high‑quality offline data, and the challenges posed by distribution shift between datasets and live environments.

Q&A Session – The speaker answered questions about defining actions and rewards, model design heuristics, suitable scenarios for RL, and observed positive impacts of RL on hybrid ranking and diversity optimization in production.