How Multimodal Large Models Boost International Product Information Understanding

Background

In AliExpress’s international e‑commerce scenario, product information (titles, images, descriptions, and attribute fields) is the core semantic bridge between sellers and overseas consumers. Missing or inaccurate attributes hinder user decisions, conversion efficiency, and overall experience.

Product publishing and selection efficiency – AI recommends missing core attributes (e.g., material, gender, season) and standardizes data for downstream analytics.

Traffic distribution optimization – fine‑grained product features and rich semantic tags improve search and recommendation relevance.

Data diagnosis and governance – a quantitative quality scoring system enables white‑box diagnostics and detection of negative content such as counterfeit or false advertising.

Key Challenges

Missing or inaccurate core attribute values due to free‑form seller input.

Incomplete Category‑Property‑Value (CPV) schema coverage, especially for long‑tail attributes.

Coarse, single‑semantic CPV representation that requires downstream tag expansion to match user language.

Algorithmic Tasks

Core attribute value back‑filling and error correction – a multimodal large model jointly processes images, titles, and descriptions to auto‑complete missing attributes and verify consistency.

Active discovery of CPV schema gaps – the model scans massive product catalogs, extracts attribute expressions not covered by the current schema, and feeds them to operators for dynamic schema evolution.

Generation of front‑end application tags – beyond basic CPV, the model predicts richer semantic tags (e.g., "commute style", "camping gear") to bridge product data with search/recommendation systems.

Technical Solution Overview

Product CPV Understanding

Attribute Extraction and Reasoning

Required capabilities:

Multimodal joint modeling – fine‑grained alignment of image and text for reliable attribute judgment.

Long‑context semantic reasoning – attributes often appear implicitly across image, title, and description, requiring contextual inference.

Open‑ended generative modeling – the model can output previously unseen attribute values (newV) without a predefined candidate set, enabling zero‑shot or few‑shot generalization.

Limitations of traditional methods:

NER & Extractive QA only extract explicitly mentioned values.

Multi‑label classification restricts output to a closed candidate list.

Retrieval‑augmented two‑stage pipelines suffer from recall bottlenecks and weaker semantic modeling.

Therefore, a generative multimodal large model (MLLM) is chosen as the backbone.

CPV Schema Gap Discovery

The pipeline consists of four stages:

Attribute prediction model inference – the trained CPV‑understanding MLLM scans the entire catalog and outputs raw predictions for each attribute.

High‑frequency candidate clustering – ambiguous or unmapped values are clustered and normalized using a general large model.

In‑library comparison – normalized candidates are matched against existing CPV lists; high‑similarity matches are marked "potentially similar", otherwise "new expression".

Category‑level operator review – operators decide whether to merge similar candidates into the schema or add new values.

Application Tag Understanding

Application tags form a decision‑factor system that shifts focus from "what the product has" to "what users care about". They connect the supply‑side CPV with consumer‑side behavior and support three goals:

User guide matching – extract key attributes that truly influence purchase (e.g., "commute style" for shoes) for tag recommendation and matching.

Category planning – use decision‑factor tags to analyze market trends, guide supply assessment, and discover business opportunities.

Fine‑grained operation – customize strategies per category (e.g., energy‑efficiency for appliances, silhouette for dresses).

Core Optimizations

Product CPV Understanding

Baseline: a generic 7B MLLM + SFT fine‑tuning with input (image, title, description, attribute list, reference values) and JSON output.

Observed bottlenecks:

Perception alignment – the visual encoder (ViT) pretrained on natural images struggles with fine‑grained e‑commerce visual cues.

Cognitive completion – text encoder lacks deep domain knowledge; noisy seller‑filled data introduces supervision noise.

Complex reasoning – multi‑step logical inference (e.g., deducing suitable age for a toy) is unreliable, leading to hallucinations.

Solution: a three‑stage MLLM collaborative training system.

Stage 1 – Continued Pre‑training for Perception

Round 1 adds an image‑captioning task on a 7B multimodal base. The caption task forces the model to describe fine‑grained visual details and align them with CPV labels. Experiments show +1.97 pt precision and +0.68 pt recall on image‑derived test subsets compared to the generic base.

Round 2 – Product Description + Direct Answer Mixing

Introduces a product‑description generation task that requires the model to produce natural‑language explanations for each attribute, combining visual observation, textual clues, and reasoning. Weak‑supervision is created by cleaning seller‑filled data with a high‑performance MLLM and distilling 72B model‑generated descriptions.

Stage 2 – Multi‑Strategy SFT for Cognitive Completion

Pure‑text LLM (llama3‑8b) is distilled to supplement the multimodal model’s language ability. Two‑phase strategy: (1) domain‑adapted SFT on attribute prediction data, (2) inference on the full training set to generate enhanced answers for further distillation. Adding image captions as extra input improves recall on text‑only answer subsets by +5.67 pt.

Knowledge Retrieval‑Augmented Generation (RAG)

Attribute‑level knowledge retrieval augments the model with explicit evidence. For each attribute, a knowledge triple (label_name, evidence_text, candidate_value) is built, indexed, and injected into the prompt as a JSON field, turning implicit reading into explicit evidence‑driven reasoning. Coverage experiments show:

When knowledge is retrieved, precision 94.44 % (+1.03 pt) and recall 76.26 % (+3.22 pt).

When not retrieved, precision 91.27 % (+1.89 pt) and recall 55.79 % (‑0.04 pt).

Stage 3 – Complex Reasoning via CoT + RL

Three‑phase evolution on Qwen2.5VL‑7B:

CoT distillation – use Qwen2.5VL‑72B to generate chain‑of‑thought outputs, filter for quality, and fine‑tune the 7B model.

Reject sampling – generate multiple CoT candidates per item, manually select correct ones, and add them to training; yields +0.68 pt accuracy and +1.39 pt recall.

Reinforcement learning (GRPO) – construct difficulty‑aware RL data, define format (0.1) and accuracy (0.9) rewards, and apply Clip‑Higher to stabilize training; overall accuracy improves by +2.44 pt.

Final Results

The three‑stage system (perception‑enhanced pre‑training, cognitive‑completion SFT, and CoT + RL reasoning) significantly raises coverage of long‑tail and complex samples, shifts the model from pattern‑matching to evidence‑driven judgment, and provides a foundation for precise downstream inference.

Technical Outlook

Unified multi‑task pre‑training to broaden CPV coverage and support additional downstream tasks (category prediction, product similarity, query relevance).

Adaptive CoT mechanisms that bypass reasoning for simple samples and compress CoT chains for efficiency.

Further automation of labeling and evaluation to reduce annotation cost.

Construction of a unified knowledge representation that merges CPV and application tags for long‑term platform value.