Vision-OPD Enables Multimodal LLMs to See Fine Details in a Single Forward Pass

Problem

Multimodal large language models (LLMs) achieve strong performance on generic image‑caption tasks but often fail when the answer depends on a tiny, decisive detail. For example, when asked “What color is the ear‑muff in the picture?” the model looks at the full image, is distracted by surrounding context, and answers “black” instead of the correct “green”. This phenomenon is called the region‑to‑global gap : accuracy on cropped, detail‑focused patches is 18–22 percentage points higher than on the whole image, even for large closed‑source models such as Gemini‑3.5‑Flash, Gemini‑3.1‑Pro, and GPT‑5.4.

Key Insight

Fine‑grained performance is limited not by object recognition ability but by the model’s ability to focus on the relevant region (“see‑and‑stay‑on‑the‑detail”). Providing a cropped, high‑resolution view during inference improves accuracy but incurs heavy inference overhead.

Vision‑OPD Framework

Training framework

Vision‑OPD creates two views from the same model:

Crop Teacher : receives a zoomed‑in crop that contains the decisive evidence.

Full‑Image Student : receives the original, unaltered image.

During training, the student generates its own answer trajectory on the full image (online sampling). For each token prefix, Vision‑OPD computes the logits of both teacher and student and minimizes their divergence, effectively transferring the teacher’s privileged local perception to the student. This process requires no external teacher, ground‑truth labels, or reward model, and adds no extra tools at inference time.

Data synthesis

A fully automated pipeline generates ~6.2 K triplets (full‑image, cropped‑image, question) from unlabeled images:

Detect and segment small‑area regions likely to hold fine‑grained evidence.

Prompt the model to formulate a question answerable by looking only at that region (e.g., “What color is the object in the red box?”).

Overlay the region’s bounding box onto the original image and add a spatial constraint (“focus on the red box”).

Crop and enlarge the region by 2× to produce the teacher input.

Because the crops are extracted automatically, the method scales to any image corpus.

Experiments

Comparison with SOTA models

Vision‑OPD was applied to Qwen‑3.5‑4B and Qwen‑3.5‑9B and evaluated on six fine‑grained visual‑understanding benchmarks: V* Bench, ZoomBench, HR‑Bench 4K/8K, and MME‑RealWorld (EN/CN). Results:

Vision‑OPD‑9B achieved an average score of 79.68, ranking first overall and surpassing closed‑source Gemini‑3.5‑Flash, Gemini‑3.1‑Pro, and GPT‑5.4.

Vision‑OPD‑9B also outperformed the much larger open‑source Qwen‑3.5‑397B.

Compared with “Thinking‑with‑Images” agents that repeatedly crop and re‑query, Vision‑OPD required only a single forward pass.

Vision‑OPD‑4B scored 77.07, beating GPT‑5.4 and Kimi‑K2.6.

Generalization (no catastrophic forgetting)

On non‑fine‑grained vision tasks (MMVP, CV‑Bench, MMStar, POPE), Vision‑OPD retained the original model’s capabilities, demonstrating “specialization without neglect”.

Gap convergence during training

Training curves show that the region‑to‑global accuracy gap steadily shrinks as training progresses, confirming that the model learns to extract fine‑grained evidence directly from the whole image.

Contributions

The work reveals that the bottleneck for fine‑grained visual understanding lies in “staying on the detail” rather than “recognizing the object”. By treating the model’s own cropped‑view performance as a privileged supervision signal, Vision‑OPD achieves:

Online sampling + token‑level distillation, avoiding the mismatch between offline prefixes and inference distribution.

Dense token‑level gradients, unlike sparse reinforcement‑learning rewards.

A reproducible, low‑cost training pipeline that requires only ~6.2 K synthetic examples.

Consequently, a 9B multimodal LLM can “see” fine details in a single forward pass, outperforming larger closed‑source models and multi‑step “thinking‑with‑images” agents while preserving general vision abilities.

Paper: https://arxiv.org/abs/2605.18740

Code: https://github.com/VisionOPD/Vision-OPD