V-STAR: A Value‑Driven Reinforcement Learning Paradigm for Generative Recommendation

Background

Generative Pre‑trained Recommender (GPR) unifies the advertising recommendation pipeline into an end‑to‑end generative framework and has been deployed at large scale. However, during reinforcement‑learning (RL) fine‑tuning, beam‑search candidate construction (probability‑driven) and GRPO policy updates (reward‑driven) operate on inconsistent objectives, causing severe degradation of the RL training signal.

Structural Mismatch in RL Fine‑tuning

Two concrete problems arise:

High‑value candidates are pruned early because their SID prefix tokens have low probability due to sparse historical interactions.

The candidate set becomes homogeneous; sibling items share high‑probability prefixes, leading to near‑zero variance in group‑wise rewards and loss of gradient signal.

Core Idea: Directed Search Budget

Observation: under a fixed decoding budget, not all generation steps benefit equally from additional exploration. Budget should be allocated to prefixes that are both high‑value and high‑uncertainty.

V‑STAR Framework

V‑STAR consists of two cooperating components:

1. VED – Value‑Guided Efficient Decoding

VED adds a lightweight Transformer + MLP on top of the GPR backbone to learn a value function estimating the expected discounted return of each prefix (TD learning). Dense step‑wise rewards are constructed by computing the cosine similarity between the ground‑truth item embedding and the weighted average embedding of all candidates sharing the prefix.

Decoding proceeds in three steps:

Learn the shallow value model.

Use a low‑overhead probability search (beam) to quickly identify promising regions and detect prematurely pruned branches.

At each prefix compute a joint priority ϕ (expected return) and θ (next‑step uncertainty); only when both are high is the branch expanded.

This yields a candidate set that covers high‑value long‑tail items while preserving intra‑group reward distinction, all within the original token budget.

2. Sibling‑GRPO

While VED ensures good candidates are present, Sibling‑GRPO addresses whether the model can learn from them. It performs relative advantage estimation within each sibling group (candidates sharing the same parent prefix) instead of global normalization. The sibling‑relative advantage is computed as the difference between a child’s average reward and the group mean, scaled by the group’s standard deviation. Joint training of GRPO and Sibling‑GRPO concentrates gradient updates on decision nodes that truly differentiate candidates.

Training–Inference Separation

VED and Sibling‑GRPO are active only during training; inference continues to use standard beam search, incurring zero additional online cost.

Experimental Results

Offline Evaluation

Using two subsets of the Amazon Review dataset (Industrial and Office Products) with Qwen2.5‑1.5B as backbone, V‑STAR outperforms all baselines (SASRec, TIGER, D3, S‑DPO, MiniOneRec) on HR@K and NDCG@K.

Online A/B Test

In the video‑account advertising scenario, V‑STAR improves GMV by 1.23% over the baseline without increasing inference latency.

Ablation Studies

Decoding Strategy : Beam Search (probability‑driven) prunes high‑value branches; Top‑K adds random exploration but lacks reward guidance; VED achieves the best trade‑off.

Priority Scoring : Joint value + entropy scoring consistently outperforms value‑only or entropy‑only across datasets; entropy alone wastes budget on uncertain yet low‑value regions, while value‑only may over‑exploit saturated high‑value nodes.

Training Objective : Standard GRPO performs worst due to global variance flattening; Sibling‑GRPO improves HR by focusing updates on critical split nodes; the joint objective (global + sibling) yields the highest HR and NDCG.

Probability‑Reward Alignment Analysis

Spearman correlation between prefix log‑probability and true reward decreases with depth, whereas the value predictor remains highly aligned across all levels, confirming that V‑STAR bridges the probability‑reward gap.

Budget Scaling

When matching decoding token budgets, VED consistently surpasses beam search, achieving 3–4× the effectiveness of beam search at the same budget.

Case Study

On the Office Products subset, a query with historical items (pencil knife, postcards, etc.) and ground‑truth ACCUTECK digital scale demonstrates that Beam Search fails to retrieve the target, Top‑K explores new branches but still misses, while VED discovers the target in a new high‑value branch, illustrating VED’s ability to break out of historical prefix constraints.

Conclusion

GPR establishes an industrial‑scale generative recommendation foundation; V‑STAR resolves the structural bottleneck in RL fine‑tuning by aligning candidate construction with reward signals through value‑guided decoding and tree‑structured credit allocation, all without extra inference cost.