Can Divide‑and‑Conquer Boost Embedding‑Based Retrieval in Recommenders?

The author shares the Kuaishou paper “Divide and Conquer: Towards Better Embedding‑based Retrieval for Recommender Systems from a Multi‑task Perspective” (arXiv:2302.02657). Although not yet deployed, the idea is described as simple, effective, and requiring additional engineering resources.

1. Sample Optimization

The industry saying “ranking is the art of models, recall is the art of samples” highlights the importance of sample selection in the recall stage, which operates over the entire item pool. Two key aspects are discussed:

Negative samples : Recall should not use only easy negatives (which fail to capture fine‑grained user interests) nor only hard negatives (which may be biased and miss items relevant to the whole pool). A balanced mix of easy and hard negatives is needed. The paper provides empirical evidence that grouping items and sampling negatives within each group improves performance.

Positive samples : Besides clicks, exposure can also serve as a positive signal. A pairwise loss can be constructed with a hierarchy click > exposure > ranking > random, as described in Alibaba’s “Multi‑Objective Personalized Product Retrieval in Taobao Search”.

2. User Multi‑Interest Vector Representation

Traditional two‑tower models encode a user with a single vector, which suffers from two problems:

A single vector struggles to capture a user’s diverse interests.

Recall based on one vector tends to produce item clustering, reducing diversity.

Industry solutions such as Alibaba’s MIND (Multi‑Interest Network with Dynamic Routing) address this by learning multiple interest vectors per user.

Paper Structure

The core idea is to apply divide‑and‑conquer: partition candidate items into K groups based on similarity, then for each positive sample draw negatives only from its own group. At inference time, recall is performed separately within each group and the results are merged.

Balancing easy and hard negatives becomes easier within groups; the paper’s experiments confirm this.

Recall per group avoids cross‑group competition, improving intra‑group recall while sacrificing some inter‑group performance. The overall system must balance both.

For user representation inside each group, three methods are described:

Inject the group ID as an additional feature when encoding the user sequence (e.g., concatenate group embedding t with behavior list s before the Transformer).

Use a multi‑task MoE (Mixture‑of‑Experts) where each group is treated as a separate task producing its own user vector.

Multiply the group embedding with the user sequence embedding, which the author finds both effective and efficient.

Future Outlook

The author suggests further experiments such as:

Adding a feature indicating which group each item belongs to in the user behavior sequence.

Treating the group ID as a target and applying target‑attention mechanisms to model interest per group.

Overall, the divide‑and‑conquer strategy offers a clear engineering path to improve embedding‑based recall by simplifying negative‑sample selection and enabling multi‑interest user representations.