MOON 3.0: Attribute Reasoning for E-commerce Multimodal Representation (ACM MM'26)

Background: Fine-Grained Attribute Matching in E-commerce Search

In e-commerce search, user clicks and purchases depend not on coarse similarity but on fine-grained attribute matches such as lace collar, bow, button style, and local decorations. Two products may share category, color, and silhouette yet differ in key details that determine distinct user preferences. Existing models are often dominated by strong signals like color and contour, overlooking weak but critical local attributes.

Three Stages of Multimodal Representation Learning

Cross-modal semantic alignment: Early dual-stream architectures projected images and text into a shared space but only aligned at the representation level, failing to capture complex multi-image, multi-text semantic structures.

Deep multimodal fusion: MLLMs as unified encoders enabled joint encoding and deep cross-modal interaction for image-only, text-only, and image-text inputs, moving from alignment to semantic fusion.

Fine-grained semantic decomposition: Leveraging MLLM generation and reasoning to explicitly parse entities, attributes, relations, scenes, and intents before representation extraction, turning representation into explicit modeling of semantic elements and their relational structure.

MOON Series Evolution

MOON 1.0 (WSDM'26): Task Unification

Introduced generative MLLM for e-commerce multimodal representation, jointly training image search, text search, and product search to build a unified semantic space. Demonstrated feasibility of one representation serving multiple tasks but suffered a seesaw effect: gains in one task often degraded another, indicating difficulty coordinating modality and task learning rhythms.

MOON 2.0 (CVPR'26): Modality Collaboration

Addressed dynamic modality balance via Modality-driven MoE, Dual-level Alignment, and Image-text Co-augmentation across architecture, training objectives, and data construction. Alleviated modality dominance and seesaw effects but still struggled to capture decisive fine-grained details when products are highly similar in category, color, and outline.

MOON 3.0 (ACM MM'26): Attribute Reasoning

Current MLLM representation methods treat the model as a feature extractor, pooling the last token or averaging all tokens to produce a global vector. This direct encoding paradigm lets dominant semantics (category, color, silhouette) overshadow weak but critical details (material, collar, buttons, patterns). MOON 3.0 shifts to reason-then-represent : first autoregressively decompose input into structured, multi-dimensional attribute sequences, then fuse original content with reasoning traces to generate unified representations.

MOON 3.0 Methodology

The pipeline processes query, positive, and negative samples (image, text, or image-text). Instead of directly outputting a global vector, the model first performs autoregressive reasoning to explicitly parse structured attribute sequences, then combines raw content and reasoning to produce the final representation. Three key modules tackle challenges introduced by attribute reasoning:

Multi-head Modality Fusion

Based on reasoning representations, dynamically fuses raw visual features, text features, and image-text interaction features according to consistency between each modality and reasoning semantics. A multi-head mechanism learns differentiated fusion weights in different semantic subspaces, allowing the model to selectively revisit raw image-text information after reasoning, supplementing details diluted during reasoning.

Joint Contrastive and Reinforcement Learning

Supervised fine-tuning (SFT) teaches attribute structure and generation format but cannot judge which attributes best discriminate positive from negative samples in retrieval. MOON 3.0 combines contrastive learning and reinforcement learning: contrastive learning directly constrains the representation space, pulling query-positive pairs closer and pushing query-negative pairs apart; reinforcement learning uses GRPO (Group Relative Policy Optimization) with rewards from output format, reasoning length, retrieval performance, and attribute quality. This synergy lets the model learn not only how to generate correct attributes but also which attributes improve retrieval discrimination.

Fine-grained Residual Enhancement (FIRE)

FIRE preserves key local information throughout the entire forward pass across visual encoding, cross-modal fusion, and language decoding:

Visual Encoding: Patch-level gated residuals in the visual encoder combine shallow local features with deep semantics to adaptively highlight product subject, texture, and design elements while suppressing background noise.

Cross-modal Fusion: Features from different visual encoder levels are projected and injected into early LLM layers, enabling multi-granularity participation of local textures and design patterns in image-text fusion.

Language Decoding: Long-range residual connections re-inject shallow multimodal cues into deep LLM representations, mitigating continuous attenuation of key image-text details during long-sequence reasoning and deep network propagation.

Experiments

MBE 3.0 Benchmark

Constructed from real e-commerce search logs, MBE 3.0 provides image, text, and multimodal attribute annotations for each product in query-positive-negative triplets. Contains 7.69 million training samples and 910 thousand high-quality test samples, supporting image search, text search, product search, product classification, and attribute prediction.

Zero-shot Evaluation

After unified representation learning, MOON 3.0 is evaluated zero-shot on MBE 3.0, M5Product, and Fashion200K without task-specific fine-tuning. Radar charts show leading performance across multimodal retrieval, product classification, and attribute prediction, indicating strong cross-task and cross-dataset generalization.

Main Results on MBE 3.0

MOON 3.0 outperforms general multimodal representations (SigLIP2, Qwen3-VL-Embedding, GME-Qwen2VL, MM-Embed) and e-commerce specialized representations (FashionCLIP, CASLIE-S, MOON 1.0, MOON 2.0). Retrieval R@10 scores: Image-to-Product 56.39%, Text-to-Product 48.10%, Product-to-Product 60.30%, Image-to-Text 42.95%, Text-to-Image 38.93%. Compared to MOON 2.0, improvements are +12.77, +10.48, and +10.71 percentage points on Text-to-Product, Image-to-Text, and Text-to-Image respectively. Classification accuracy reaches 86.40% (+19.13), attribute prediction 49.92% (+13.56).

Cross-dataset Generalization

On M5Product and Fashion200K, MOON 3.0 also achieves leading results, confirming robustness across data distributions.

Ablation Study

Removing each component shows its contribution:

Remove Reasoning: Largest drop; Image-to-Text R@10 falls from 42.95% to 17.12% (-25.83), classification from 86.40% to 57.52% (-28.88), attribute prediction from 49.92% to 34.21% (-15.71). Confirms explicit attribute reasoning as core foundation.

Remove Multi-head Modality Fusion: All five retrieval tasks decline; Text-to-Image R@10 drops from 38.93% to 31.95% (-6.98). Shows adaptive raw signal supplementation mitigates information dilution in long reasoning chains.

Remove Reinforcement Learning (GRPO): Attribute prediction falls from 49.92% to 41.54% (-8.38), multiple cross-modal retrieval metrics drop. Validates RL's role in breaking SFT imitation bottleneck.

Remove FIRE: Classification drops to 80.33% (-6.07), attribute prediction to 42.22% (-7.70), retrieval also degrades. Demonstrates importance of preserving fine-grained signals throughout the forward pass.

Visualization Analysis

Qualitative examples on product-to-product retrieval show MOON 3.0 extracts structured attributes (e.g., for an insulated tumbler: category "Insulated Tumbler", colors "Beige, Brown", design element "Dog Motif"). Top-ranked candidates match key attributes; lower-ranked candidates diverge in color and pattern. Similar behavior observed for phone cases (model, pattern theme) and hoodies (splicing design, collar style). Intermediate reasoning provides explainability: why a product ranks high, not just a similarity score.

Conclusion and Future Work

Summary

MOON 1.0: Unified heterogeneous tasks via generative MLLM, validated one representation for multiple tasks (WSDM'26).

MOON 2.0: Dynamic modality balance via Modality-driven MoE, Dual-level Alignment, Image-text Co-augmentation, mitigating modality dominance and seesaw effects (CVPR'26).

MOON 3.0: Reasoning-aware multi-granularity semantic representation via Multi-head Modality Fusion, Joint Contrastive and Reinforcement Learning, and FIRE, moving from implicit global encoding to explicit fine-grained reasoning (ACM MM'26).

Outlook

LLM-driven automated data engine: Use MLLM for end-to-end sample generation, quality judgment, and error attribution: generate multimodal queries, positives, and hard negatives with attribute conflicts; mine real query-product relationships from massive logs; filter false negatives and noisy labels via MLLM-Judge, confidence calibration, and human spot-checks; feed model bad cases, training loss, and uncertainty back to data side for targeted supplementation of long-tail categories, fine-grained attributes, and complex intents, forming a continuous data-production–model-training–evaluation–diagnosis–data-feedback loop.

Deep integration of perception, reasoning, and generation: Beyond attribute reasoning, combine product understanding with user intent. Model should not only recognize what a product is and its attributes, but also understand which conditions users truly care about, judge which needs are satisfied, which details deviate, and what key differences exist among candidates. This shifts from answering "what is the product and how similar are they" to understanding "what does the user want, why does the product match, and where exactly do products differ", building a reasoning-oriented representation foundation for e-commerce search.