Preference-oriented Diversity Model Based on Mutual Information for E-commerce Search Re-ranking (SIGIR 2024)

The paper, accepted at SIGIR 2024, introduces PODM‑MI, a Preference‑oriented Diversity Model based on Mutual Information, designed to balance accuracy and diversity in JD e‑commerce search re‑ranking.

Traditional re‑ranking methods improve scoring precision at the cost of result diversity, while diversity‑focused approaches often degrade relevance; PODM‑MI addresses this trade‑off by jointly modeling user preference and item diversity.

PODM‑MI employs variational inference with a multivariate Gaussian distribution to capture uncertain user diversity preferences and candidate item representations. It maximizes the mutual information between these distributions to enhance their correlation, producing a utility matrix that adaptively re‑orders items according to individual user intent.

User Preference Modeling (PON): The model incorporates historical queries, session behavior, and query streams to represent evolving user intent. Preferences are encoded as probability distributions (mean vector and diagonal covariance), enabling dynamic and uncertain modeling of user needs.

Similarity Alignment Module (SAM): Mutual information is used as a metric to quantify the shared information between user preference and candidate items. A variational posterior estimator provides a tractable lower bound for the mutual‑information objective, allowing effective optimization.

The overall loss combines a PRM classification loss and a mutual‑information loss, as shown in the following equations:

Extensive online A/B testing on JD’s main search engine shows that PODM‑MI improves user purchase probability (UCVR) and increases result diversity. Visual analyses using entropy metrics and T‑SNE clustering illustrate clear separation of user intent groups, confirming the model’s ability to adapt rankings to diverse or focused user goals.

Future work includes incorporating finer‑grained features for better buy‑vs‑browse intent modeling, further optimizing user intent updates, and making intent modeling explicitly influence ranking decisions.

The article concludes with a recruitment notice from JD’s search algorithm team, inviting candidates for social‑recruitment and internship positions related to large‑model‑based generative retrieval and ranking.