Decoupled Multimodal Fusion (DMF) Boosts User Interest Modeling for CTR Prediction

1. Background

Multimodal content (text, images, etc.) provides rich semantic cues that can complement sparse ID features in recommendation systems, but industrial pipelines typically adopt a two‑stage approach: offline extraction of multimodal embeddings followed by downstream ID‑based models. Directly fusing raw multimodal embeddings with ID features suffers from a large distribution mismatch, leading to sub‑optimal performance.

Existing works align target items with user history via multimodal similarity histograms or attention‑based methods, but they follow a modality‑centric modeling strategy that processes ID and multimodal signals independently, limiting fine‑grained interaction.

2. DMF Framework

DMF proposes a decoupled multimodal fusion architecture that combines two modeling paradigms:

Modality‑enriched modeling – implemented by the Decoupled Target Attention (DTA) network, which enables fine‑grained interaction between content semantics and behavior signals while preserving efficiency.

Modality‑centric modeling – captured by a similarity‑histogram (SH) module.

The DTA module computes target‑aware cosine similarity scores between the target item and each historical item, discretizes these scores into embeddings, and feeds them to a multi‑head target attention mechanism. The SH module directly consumes raw similarity scores. Both representations are fused by the Complementary Modality Modeling (CMM) component, which balances semantic generalization (modality‑centric) and behavior personalization (modality‑enriched).

3. DTA: Balancing Efficiency and Effectiveness

DTA treats the target‑aware similarity as side information and feeds it together with ID‑based embeddings into a multi‑head target attention block (early fusion). Early fusion offers strong expressive power but incurs heavy repeated computation for each candidate item. Late fusion postpones the combination to the final prediction layer, reusing ID‑based computations across candidates but losing fine‑grained interaction.

Decoupled fusion (DTA) separates the target‑aware computation from ID‑based computation, allowing the latter to be reused across candidates while mapping similarity scores via discretization + embedding lookup, thus avoiding costly linear transforms. Theorem 1 (proved in the paper) states that for any ε > 0, with enough histogram buckets and proper initialization, DTA’s output can be made within ε of the early‑fusion model’s output.

Empirically, DTA achieves a three‑fold single‑machine throughput increase on NVIDIA A10 GPUs compared with early fusion, with negligible loss in predictive power.

4. CMM: Combining Generalization and Personalization

While DTA handles fine‑grained interaction, different user activity levels require different modeling emphasis. Low‑activity users benefit from semantic generalization, whereas high‑activity users need personalized behavior modeling. CMM fuses the modality‑centric and modality‑enriched interest vectors using a weighted sum (weight α is a hyper‑parameter). Experiments show optimal α = 0.3 on the Amazon dataset (dominant low‑activity users) and α = 0.7 on the Lazada industrial dataset (dominant high‑activity users), suggesting a dynamic weighting mechanism could further improve “one‑size‑fits‑all” personalization.

5. Experiments

5.1 Model Comparison

Baseline ID‑only models (SASRec, DIN, TA) and multimodal baselines (SIMTIER, MAKE) were compared against DMF. DMF consistently outperformed SOTA multimodal baselines on both public (Amazon) and industrial (Lazada) datasets in terms of AUC/GAUC.

5.2 Ablation Study

Key findings:

Early‑fusion methods achieve similar AUC/GAUC to DTA but are impractical for full deployment due to poor efficiency.

Late‑fusion methods lag behind DTA by 0.30%/0.13% in AUC/GAUC.

Removing the multimodal enhancement on the Value (V) side of DTA reduces AUC/GAUC by 0.21%/0.29%, confirming the importance of enhancing both Key (K) and Value (V).

DTA offers the best trade‑off between efficiency and effectiveness.

5.3 Case Study

In a real example, a user’s history includes high‑frequency clicks on “cotton fabric” and medium clicks on “teak display cabinet”. The baseline TA model assigns the highest attention weight (0.2965) to “cotton fabric” due to popularity, while the multimodal similarity between the candidate “assembled teak cabinet” and “teak display cabinet” is 0.9243, yet TA only gives it 0.2127 weight. DTA incorporates the high similarity as side information, raising the attention weight for the candidate to 0.6586, demonstrating its ability to overcome popularity bias.

5.4 Online A/B Test

DMF was deployed in Lazada’s cross‑shop recommendation on product detail pages, using TA+SIMTIER as the baseline. Over a 12‑day test on the TH site, DMF achieved a 5.3% lift in click‑through conversion rate (CTCVR) and a 7.43% increase in gross merchandise volume (GMV), with no noticeable latency increase. The DTA component alone delivered a three‑fold throughput improvement on a single machine.

6. Conclusion and Outlook

The paper presents DMF, a decoupled multimodal fusion framework that addresses the efficiency challenges of target‑aware feature integration while preserving strong expressive power. DTA provides an efficient yet effective fusion, and CMM balances semantic generalization with fine‑grained personalization. Future work includes end‑to‑end training of multimodal features together with the prediction model to further align semantic and behavioral spaces.