MMQ Advances Multimodal Fusion and Aligns Behavior for Large-Scale Recommendation Models

Background

Recent scaling‑law observations for large language models (LLMs) show that increasing parameters and data yields predictable performance gains, inspiring similar investigations in recommendation systems. Industrial‑grade deep‑learning recommendation models (DLRMs) process billions of user actions daily and rely on massive, heterogeneous feature vocabularies. Unlike the homogeneous token vocabularies of LLMs, DLRM vocabularies are large, sparse, and heterogeneous, limiting linear performance scaling with compute.

Semantic IDs, which decompose massive items into multiple semantic tokens, improve generalization and storage efficiency but still face bottlenecks in multimodal fusion and alignment with recommendation tasks.

Challenges of Existing Semantic‑ID Methods

Insufficient multimodal representation: current methods either fuse modalities into a single vector, losing modality‑specific information, or generate separate IDs per modality, failing to capture cross‑modal synergy.

Misalignment between content and behavior spaces: semantic IDs are learned in a multimodal content space, while recommendation tasks operate in a user‑behavior space, causing a gap that can introduce noise and degrade downstream performance.

ID‑collision: compressing millions of items into thousands of semantic IDs creates many‑to‑one mappings, harming the independence of head items that dominate exposure and clicks.

MMQ: Multimodal Mixture‑of‑Quantization Framework

MMQ addresses the above challenges with a two‑stage pipeline: (1) a multimodal shared‑specific expert training stage and (2) a behavior‑aware fine‑tuning stage.

1. Multimodal Shared‑Specific Expert

Item multimodal features (e.g., title, description, image) are fed into a network comprising shared experts that capture modality‑common knowledge and specific experts that model modality‑unique characteristics. Orthogonal constraints are applied to expert parameters to encourage low‑redundancy latent embeddings. A cosine‑based quantizer maps latent embeddings to codebooks, emphasizing vector distance rather than magnitude for stable quantization.

2. Behavior‑Aware Fine‑Tuning

Soft Indices connect the tokenizer to downstream recommendation tasks. During fine‑tuning, cosine similarity between latent embeddings and all codebook vectors yields a soft index that is used for forward lookup; gradients from the recommendation loss are back‑propagated through a Straight‑Through Estimator (STE) to adjust the tokenizer. A reconstruction loss is retained to preserve semantic fidelity.

Experimental Validation

MMQ was evaluated on Lazada production data and public benchmark datasets for both recall and ranking tasks. Compared with state‑of‑the‑art multimodal semantic‑ID methods, MMQ consistently achieved lower reconstruction loss and higher downstream recommendation accuracy.

Recall Evaluation

Ranking Evaluation

Ablation studies showed that removing the shared expert degrades performance even when increasing the number of specific experts, confirming the efficiency of the shared‑specific design.

Scalability

Varying the semantic‑ID length from 6 to 18 tokens demonstrated a clear positive correlation: longer IDs improved NDCG@5 and NDCG@10 without increasing reconstruction loss or token entropy, indicating that MMQ scales effectively with token count.

Online A/B Test

MMQ was deployed in Lazada’s generative recall pipeline, serving 10% of random traffic. Compared with the baseline RQ‑VAE, MMQ yielded statistically significant online improvements: REV +1.29%, CVR +4.33%, GMV +2.61%, and ROI +1.18%.

These results confirm MMQ’s practical value and deployment readiness.