Multimodal Content Understanding in Baidu Commercial Systems: The ViCAN Model and Its Applications

The presentation introduces the background and challenges of multimodal content understanding in Baidu's commercial advertising system, where rich media such as images and videos have replaced pure text and require models that capture both scene‑specific differences and common user intents.

To address these issues, Baidu built a unified multimodal pre‑training model called ViCAN. The model leverages large‑scale image‑text pairs (≈100 billion) collected from Baidu Image Search and commercial ad data, and is trained with both coarse‑grained and fine‑grained contrastive learning as well as a multimodal masked language modeling task.

ViCAN’s architecture consists of dual 48‑layer Transformers for text and vision, with contrastive loss that aligns CLS tokens across modalities and additional token‑level alignment between image patches and text tokens. The masked language task uses cross‑attention to predict masked text tokens with visual context, improving cross‑modal reasoning.

Integration of ViCAN into Baidu's ad pipeline enhances three key stages: recall, creative selection, and ranking. In recall, multimodal triggers replace text‑only triggers and multimodal features are added to user‑behavior graphs using a two‑layer domain‑aware aggregation. In the creative stage, ViCAN upgrades image‑text relevance filtering and replaces material‑ID features with continuous multimodal embeddings, improving material freshness and quality. In ranking, dense multimodal features are discretized via the MMDict method, mapping them to sparse IDs that can be combined with other categorical features, yielding higher AUC and better long‑tail performance.

The article also describes the MMDict quantization pipeline, which uses a shared transformer encoder followed by multi‑stage quantization (coarse, residual, product quantization) to produce hierarchical discrete IDs (10⁴‑10⁶ scale) that preserve semantic granularity while remaining efficient for large‑scale recommendation models.

Finally, the authors discuss future directions with AIGC: using ViCAN for image‑to‑text generation to produce fine‑grained descriptions that guide text‑to‑image diffusion models, creating a data‑flywheel that continuously improves both generation and understanding capabilities. A Q&A section clarifies model availability, evaluation metrics, and data‑cleaning practices.