Optimizing Individual Diversity in Recommendation Systems: Architecture, MMR and DPP Implementation at 58 Tribe

Background – In recommendation systems, diversity is a crucial metric alongside relevance, yet the two often conflict. The article explores how to balance diversity and relevance from a business perspective, using diversity as a means to boost overall performance indicators.

Significance and Challenges – The main difficulties include the vague optimization target for diversity (since it is a set‑level statistic rather than a direct user behavior) and the trade‑off between business KPIs (e.g., conversion rate) and diversity metrics.

System Architecture – The recommendation pipeline is divided into three layers: recall, rule, and diversity (re‑ranking). Optimizations are applied at each layer to ensure a diverse candidate pool while preserving relevance.

Recall‑Layer Diversification – Multi‑path recall is employed, adding diverse strategies such as personalized topic/author recall, long‑tail and temporal recall, and attribute‑based collaborative recall. These measures increase topic coverage by ~120% and category coverage by ~100% in the coarse‑ranking pool.

Rule‑Layer Diversification – Before fine‑ranking, candidates are bucketed by type and diversified within each bucket, improving topic coverage by ~80% and category coverage by ~70%. Online experiments show notable CTR and UV‑CTR gains.

Re‑ranking (Diversity) Layer – Two main algorithms are deployed: Maximal Marginal Relevance (MMR) and Determinantal Point Process (DPP). Both are tuned using three business metrics: pvCTR, uvCTR, and avgPV.

MMR Principle – MMR selects items greedily to maximize a weighted combination of relevance and novelty: score = λ * sim(query, item) - (1-λ) * max_{selected} sim(item, selected) . The implementation uses a flowchart (shown in the original figures) and adjusts the weight λ to balance relevance and diversity.

DPP Principle – DPP treats re‑ranking as a subset selection problem, modeling the probability of selecting a set Y as proportional to the determinant of a kernel matrix K: P(Y) ∝ det(K_Y) . The kernel captures item similarity, and a custom distance metric is used to handle heterogeneous item types.

Implementation Details – The Java matrix library EJML is used for efficient computations. Custom distance functions are defined to improve interpretability and business alignment, including:

Jaccard/Hamming distance: S_ij = (1 + ⟨f_i, f_j⟩) / 2

Tree‑model distance: hierarchical decay based on business taxonomy.

For DPP, three solution strategies were evaluated: (1) a window‑based re‑ranking with direct determinant computation, (2) a generic MAP inference with high latency, and (3) an incremental greedy algorithm (proposed by Huhu Video) achieving ~4 ms latency for 100 items.

Experimental Results – Parameter sweeps show that increasing diversity improves pvCTR, uvCTR, and avgPV up to an optimal point (≈0.7 for the diversity weight). Compared with the original heuristic, MMR yields +3.4% pvCTR, +5.4% uvCTR, +4.2% avgPV, while DPP achieves +5.8% pvCTR, +7.9% uvCTR, +6.0% avgPV.

Conclusion and Outlook – By integrating diversity at multiple pipeline stages and customizing similarity measures, the system successfully boosts business metrics. Future work includes exploring learning‑based diversity methods and reinforcement‑learning approaches.