UniNote: Unifying Multimodal Representation and Ranking in a Single Embedding Model

Problem and Motivation

In industrial‑scale multimodal Item‑to‑Item retrieval, a note may contain an image, a title, body text, and OCR lines. Traditional pipelines separate embedding and ranking into two stages, causing high latency and difficulty balancing accuracy with compute cost.

UniNote Overview

UniNote is a unified multimodal embedding model that merges representation learning and ranking into a single forward pass. It adopts a progressive two‑stage training paradigm: first, Supervised Fine‑Tuning (SFT) to strengthen embeddings; second, Reinforcement Learning (RL) to directly optimize ranking quality. Matryoshka Representation Learning (MRL) enables elastic multi‑granular embeddings.

Core Design

1. Unified Representation : A multimodal large language model (MLLM) serves as the backbone, compressing image‑text‑OCR data into a single vector. The model supports various Item‑to‑Item tasks such as Atomic Alignment (I2T, T2I), Subordinate Retrieval (I2Note, T2Note), Semantic Extraction (Note2I, Note2T), OCR Perception (OCR2Note, I2OCR, OCR2I), and Content Relevance (Note2Note).

2. Unified Architecture : The traditional “embedder + reranker” pipeline is replaced by a single embedding pass, eliminating the need for model cascades.

3. Elastic Deployment : MRL’s “Russian‑doll” mechanism jointly optimizes losses for multiple embedding dimensions (64, 512, 1024, 4096), ensuring high‑quality semantic discrimination even when vectors are truncated.

SFT Stage – Building Strong Embeddings

The first stage converts the MLLM from a generative model to a high‑quality embedder. Unlike CLIP’s dual‑tower, the MLLM fuses modalities in a single transformer and uses the last token as the representation. Three key techniques are applied:

Modality Replacement to Prevent Shortcut Learning : During local‑global retrieval training, the query image in a note is replaced by a semantically equivalent text description, forcing the model to rely on high‑level semantics rather than pixel similarity.

Hard Negative Mining at Multiple Granularities : Similarity‑based filtering selects candidate negatives, followed by item splitting to avoid overlap. Rule‑based counterfactual hard negatives are generated by removing target elements from positive samples, creating highly overlapping yet incorrect negatives.

Jensen‑Shannon Divergence Loss : Replaces KL divergence to provide symmetric optimization and training stability, following designs such as UniME‑v2.

RL Stage – Directly Optimizing Ranking

The second stage introduces reinforcement learning because contrastive learning optimizes global vector space but neglects fine‑grained ordering. RL treats ranking as a combinatorial optimization problem, using a reward that evaluates the entire ordered list.

Reward Design : A hierarchical reward combines penalties for irrelevant notes and rewards for relevant ones, plus absolute‑position and relative‑order components that capture both position accuracy and score ordering. The reward is monotonic, as verified by random‑ordering experiments.

Data Construction : For Item‑to‑Item tasks, each note is split into two non‑overlapping parts A (query) and B (candidate pool). B is combined with varying numbers of images to create a relevance‑graded list, then mixed with noise notes, providing high‑quality ranking supervision without manual labeling.

Algorithm : Group Relative Policy Optimization (GRPO) is employed to optimize the hierarchical reward.

Experiments

UniNote was evaluated on several Item‑to‑Item tasks defined on the Xiaohongshu platform. Compared with strong baselines RzenEmbed and Qwen3VL‑Embedding‑8B, UniNote consistently outperformed them on atomic alignment, subordinate retrieval, semantic extraction, and content relevance. The OCR‑perception task showed modest gains, attributed to the model’s emphasis on high‑level semantics over fine‑grained character recognition.

Metrics such as R@1, R@5, and P@5 were reported. For Note2Note, a multi‑label recall (R*@1) was used due to non‑unique positive candidates.

Ablation Studies

Hard Negative Mining : Moving from random negatives to the full multi‑granular hard‑negative strategy yielded up to 23.3% improvement, confirming the importance of challenging negatives. Adding rule‑based counterfactual negatives further improved performance.

Two‑Stage Training : Incorporating the RL stage increased R@5 by 1.4% and P@5 by 3.4%, demonstrating the benefit of direct ranking optimization.

MRL Compression Effect

Four embedding dimensions (64, 512, 1024, 4096) were tested. Performance generally increased with dimension size. At 512 dimensions, most tasks approached full‑dimensional (4096) performance; at 64 dimensions, performance dropped but retained about 70% of the retrieval capability on image‑text alignment, subordinate retrieval, and semantic extraction, indicating high information density.

Conclusion and Future Work

UniNote presents a novel unified embedding paradigm that fuses multimodal representation learning with ranking optimization, offering a low‑latency solution for industrial Item‑to‑Item retrieval. Future directions include extending the approach to Any‑to‑Any retrieval scenarios and improving OCR sensitivity with targeted datasets.