How We Boosted Product Pair Recommendations with LLM Scoring and BERT Distillation

1 Background

The homepage of ZuanZuan recommends items by mixing interest‑based products with exploratory items; exploratory categories are derived from related interest categories. During re‑ranking, cross‑category pairwise combinations are built and a model is trained to learn reasonable product pairings, improving exploration efficiency and user experience. The process has two stages: (1) pair collection and large‑model annotation, and (2) fine‑tuning Qwen‑7B and distilling its knowledge into BERT to address the latency of deploying large models.

2 Data Preparation

2.1 Pair Collection

Product‑level pairs are extracted from user historical behavior and pre‑processed to obtain pairs of varying frequencies.

2.2 Pair Processing

Pairs are divided into three frequency categories:

High frequency: common user preferences, manually inspected and cleaned.

Medium frequency: moderate confidence, validated by a large model, then manually inspected and cleaned.

Low frequency: long‑tail categories with sparse interactions; reasonable pairs are manually selected, augmented via few‑shot learning with a large model, and finally cleaned.

The overall workflow is illustrated below:

3 LLM Annotation Scoring

3.1 Evaluation Criteria

Compatibility is evaluated across five dimensions: category, color, brand, condition, and model. Each dimension receives a score from 1 to 10, weighted between 1% and 100% according to real‑world importance; the weighted sum yields a final compatibility score. For example, stronger category relationships receive higher scores.

3.2 Prompt Splitting Design

Long prompts cause excessive latency when scoring with a large model, so attributes are split and processed in parallel. An example prompt for the category attribute is shown below.

4 Model Training

To overcome the deployment latency of large models, knowledge is distilled into a BERT model for online use.

4.1 Sample Definition

Each pair receives a score between 1 and 10. Medium and low frequency pairs inherit scores from the LLM annotation, while high‑frequency pairs are assigned a default score of 10.

4.2 Fine‑tuning Qwen‑7B

Before distillation, the collected samples are used to fine‑tune Qwen‑7B, enabling the model to capture domain‑specific product pairing knowledge.

4.3 BERT Distillation

The distillation framework uses the fine‑tuned Qwen‑7B as the Teacher and a 340M BERT model as the Student. The loss consists of three parts: (1) distill loss aligning the student to the teacher (gradient‑free), (2) task loss for the pairing relationship, weighted by a temperature parameter, and (3) mse loss .

5 Summary and Outlook

After deployment to the homepage feed, the strategy increased PV‑detail page reach by 6% and UV‑detail page reach by 5%.

Large‑model annotation significantly reduced manual effort, enabling large‑scale sample generation.

BERT distillation made it feasible to apply large‑model knowledge online.

The current product attributes are coarse but broadly applicable; future work will explore finer‑grained attributes for better pairing optimization.

Future experiments will target new users who lack interest clicks, presenting multi‑item joint pairings of non‑interest products.