How Qwen3‑VL Embedding and Reranker Set New SOTA in Multimodal Retrieval

Background and Motivation

Internet content now includes images, short videos, scanned documents, and live streams, making traditional text‑only search engines inadequate for cross‑modal queries such as "search by image for text" or "search by video for products". After CLIP, the community has been seeking a single model and vector space that can handle all modalities end‑to‑end.

Qwen3‑VL Series Overview

The Qwen3‑VL family introduces two core models:

Qwen3‑VL‑Embedding : a bi‑encoder that provides unified embeddings for text, images, video, and documents. It comes in 2B and 8B parameter versions and supports up to 32 K tokens.

Qwen3‑VL‑Reranker : a cross‑encoder that performs fine‑grained re‑ranking. It also has 2B/8B variants and the same token limit.

One‑sentence memory: the Embedding model handles the "screening" stage, while the Reranker handles the "final" stage.

Understanding the Unified Vector Space

All modalities are projected into a shared manifold, enabling direct similarity comparison across text, images, video, and visual documents.

Technical Highlights

3.1 Multi‑Stage Training Pipeline

Stage‑0 – Contrastive Pre‑training : 2 billion synthetic image‑text pairs are used to warm‑up a base model.

Stage‑1 – Multi‑Task Fine‑tuning : High‑quality human‑annotated data are added to mitigate task imbalance.

Stage‑2 – Knowledge Distillation : The Reranker’s fine‑grained signals are fed back to improve the Embedding; a weighted merge with Stage‑1 yields an unbiased Stage‑3 model.

3.2 Matryoshka Representation Learning & Quantization

Matryoshka Representation Learning : The model is trained to produce embeddings at multiple dimensions (32, 128, 512, 1024, …); at inference time any desired dimension can be selected.

Quantization‑Aware Training : INT8 quantization incurs almost no performance loss, while binary quantization reduces memory footprint by eight times, suitable for massive indexes.

Data Engineering: Synthesizing 1 B Multimodal Pairs

Alibaba first uses Qwen3‑VL‑32B to label 20 million raw images and videos, applies quality filtering, then automatically generates query‑document‑label triples via task‑level prompts, performs hard‑negative sampling, and finally produces 300 million synthetic multimodal pairs, creating a self‑reinforcing data flywheel.

Experimental Results

5.1 Multimodal Benchmark (MMEB‑V2)

Across 78 datasets and 9 task categories, Qwen3‑VL‑Embedding‑8B achieves an average score of 77.8, ranking first and surpassing the previous best open‑source model by 6.7%.

5.2 Pure‑Text Evaluation

Qwen3‑VL‑Embedding‑8B scores 67.9 on the MMTEB multilingual benchmark, only 3 points below the dedicated text‑only Qwen3‑Embedding‑8B (70.6), demonstrating that multimodal training does not sacrifice textual capability.

5.3 Reranking Performance

Qwen3‑VL‑Reranker‑8B improves the average score by +4.1 points, raising the combined system from 73.4 to 79.2 and outperforming strong baselines such as jina‑reranker‑m0.

Conclusion and Future Directions

The combination of large models, massive data, and extensive engineering pushes multimodal retrieval to a new state‑of‑the‑art while remaining deployment‑friendly (adjustable dimensions, quantization) and preserving text performance. Future work includes adding new modalities (audio, 3D, time‑series sensors), handling longer videos (>10 min) with sparse sampling and memory mechanisms, compositional retrieval across text‑image‑audio, and lightweight edge solutions with fewer than 1 B parameters.

References and Resources

https://arxiv.org/pdf/2601.04720
Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Ranking
https://huggingface.co/collections/Qwen
https://github.com/QwenLM/Qwen3-VL-Embedding