How RAD‑DPO Aligns Preferences in OxygenSearch Generative Retrieval (SIGIR 2026)

Background

Generative retrieval (GR) in e‑commerce transforms massive SKU catalogs into structured semantic IDs (SIDs) and treats retrieval as an auto‑regressive generation task. SIDs are usually built with hierarchical clustering (e.g., RQ‑VAE or RQ‑Kmeans), enabling millisecond‑level response via beam search over billions of items.

Training typically consists of three stages: semantic‑alignment pre‑training, instruction fine‑tuning (SFT), and preference‑optimization (DPO/RL).

Why DPO?

OxygenSearch receives rich implicit feedback signals such as exposure, click, add‑to‑cart, and purchase. Traditional ranking models absorb these signals with cross‑entropy or pairwise losses, but after SFT the GR model still needs to align with user intent. Compared with other reinforcement‑learning methods (GRPO, PPO, DPO), DPO does not require an extra reward model, making it simple and low‑cost.

Special Challenges in GR

Three unique issues arise in generative retrieval:

Shared Prefix Gradient Conflict : SIDs share hierarchical prefixes; standard DPO penalizes negative samples indiscriminately, causing oscillation of correct prefix paths.

Noise Sensitivity (Pseudo‑negatives) : "Not clicked" does not always mean irrelevant; treating such items as hard negatives can distort semantic representations.

Probability Squeezing : A query often has multiple relevant positives; suppressing negatives can compress the probability space of long‑tail positives.

Algorithm Evolution: Preference‑Alignment Methods

Survey of DPO‑type algorithms shows a trend from simple pair‑wise sequence‑level optimization toward low‑cost, multi‑sample, token‑level, noise‑robust, prefix‑aware, structure‑aware methods. RAD‑DPO extends this trend to the SID structure and pseudo‑negative problem.

RAD‑DPO Core Technical Solution

Session‑Level Multi‑Label Contrast (MLGC)

In e‑commerce a query may correspond to several clicked or purchased items. Instead of a single winner, RAD‑DPO treats all positive feedback items in a session as a set and contrasts them against a set of negatives (un‑clicked items and sampled candidates). This alleviates the probability‑squeezing effect of pair‑wise DPO.

The loss adds a global SFT term that boosts the probability of every positive item in the session, and a preference loss that creates a margin between the positive set and the negative set.

Token‑Level Gradient Detachment (TLGD)

SIDs have hierarchical tokens: early tokens represent coarse categories, later tokens specify the exact product. Standard DPO penalizes the entire negative sequence, also suppressing correct shared prefixes. TLGD computes the longest common prefix length k between positive and negative sequences, then stops gradient propagation on the shared‑prefix tokens, applying penalty only after the divergence point.

Thus the forward pass still includes the full negative likelihood, but the backward pass detaches gradients on the prefix, protecting the learned hierarchical semantics.

Dynamic Reward Weighting (RDRW)

Implicit feedback contains noisy "pseudo‑negatives". RAD‑DPO measures similarity between positive and negative samples using the cosine similarity of their EOS hidden states. A smooth weighting function (based on a warm‑up distribution of similarity scores) reduces the penalty for negatives that are close to positives, mitigating over‑penalization of noisy samples.

Overall Preference Loss

The final loss combines the MLGC term, the TLGD‑processed negative log‑likelihood, and the RDRW weighting, explicitly controlling which tokens are penalized and how strongly each negative sample contributes.

Efficient Training: Label Packing

Session‑level training would naively repeat the prompt encoding for each candidate, causing large memory waste because prompts are long in e‑commerce. RAD‑DPO packs all candidates into a single unified sequence and uses a block‑diagonal attention mask so that the prompt attends to all candidates while candidates do not attend to each other. Position IDs are aligned across candidates to avoid positional bias.

Experimental Analysis

Experiments on a 30‑million‑sample offline dataset show that RAD‑DPO consistently outperforms SFT and other DPO variants on recall and MRR metrics. The advantage grows with model size (0.6 B → 8 B) and remains significant even with only 10 M training samples, demonstrating strong data efficiency.

Engineering benchmarks indicate roughly a 50 % reduction in GPU memory (no reference model) and a 3× increase in throughput thanks to label packing.

Conclusion and Outlook

RAD‑DPO provides a practical, industrial‑scale solution for generative retrieval in e‑commerce, addressing shared‑prefix conflicts, noisy implicit feedback, and multi‑positive probability squeezing. Future work includes incorporating relevance samples, joint SFT‑DPO training, and finer‑grained token‑level weighting.