How JD’s PODM‑MI Model Boosted E‑commerce Search Diversity and Sales

Connecting Technology and Everyday Life at JD

At JD, technology is portrayed not as cold code but as a bridge linking consumers to a better life. Algorithm engineer Hui Mu leverages large models for intelligent recommendation, search, and guidance, publishing four top‑conference papers and eight patents during his tenure.

From Theory to Practice: Seeking the "Best‑Fit" Solution

Transitioning from campus to industry revealed a shift from seeking the "optimal" solution to the "most suitable" one, especially in JD’s complex e‑commerce environment where user decision stages, ecosystem health, and billion‑scale traffic impose unique constraints.

Rethinking Search Ranking: The PODM‑MI Framework

Traditional search ranking often over‑exposes popular items, sacrificing long‑tail exposure. JD’s first project on main‑site search highlighted the need to adapt ranking to users’ decision stages—browsing versus buying.

The proposed three‑layer PODM‑MI re‑ranking framework addresses this:

Layer 1: Gaussian Preference Modeling —captures dynamic user preference by adjusting covariance; narrower covariance (e.g., "dress → floral dress → blue floral dress") raises accuracy weight, while broader covariance (e.g., "phone → Switch → range hood") raises diversity weight.

Layer 2: Mutual‑Information Lower‑Bound Optimization —maximizes mutual information to align diversity with user preference, presenting related items without irrelevant noise.

Layer 3: Utility‑Matrix Fusion —dynamically balances product relevance and diversity during ranking.

Field tests showed significant gains in the UCVR metric and an annual order increase exceeding ten million, earning a SIGIR top‑conference paper.

Discovering the Hourglass Bottleneck in Generative Retrieval

While advancing generative search‑recommendation, JD identified a "sand‑glass" distribution in semantic identifier (SID) codes generated by RQ‑VAE: dense clustering in the middle layer caused low code‑table utilization and hindered model training.

The root cause lies in RQ‑VAE’s hierarchical clustering: the first layer yields uniform distribution, the second layer’s residuals become highly polarized (most data near cluster centers, few far out), and the third layer re‑uniformizes, amplifying the long‑tail effect in e‑commerce data.

Two lightweight remedies were proposed:

Remove bottleneck nodes from the middle layer after full SID generation, alleviating long‑tail concentration.

Introduce an adaptive threshold to dynamically prune overly dense high‑frequency nodes in the second layer, preserving overall distribution stability.

Experiments demonstrated that pruning a proportion of high‑frequency nodes significantly improves offline recall, helping users discover desired products faster.

Broader Reflections

The experience underscores that true technical value lies in systematic solutions that bridge business needs and engineering capabilities, turning code into a “love letter” that makes shopping more efficient and enjoyable.

Related papers:

A Preference‑oriented Diversity Model Based on Mutual‑information in Re‑ranking for E‑commerce Search

Breaking the Hourglass Phenomenon of Residual Quantization: Enhancing the Upper Bound of Generative Retrieval