Why VLMs Miss Reference Images and How RefCaptioner Aligns Them Precisely

Current video‑language models (VLMs) can generate fluent captions but often fail to correctly associate candidate reference images with the corresponding video semantics, especially when many reference images are provided.

Motivation

The authors observe that as the number of reference images grows, VLMs’ ability to match video subjects with the right images drops sharply, revealing a blind spot in standard captioning paradigms that only require textual fidelity.

RefCaptioner Design

RefCaptioner builds on Qwen3‑VL‑8B‑Instruct and introduces post‑training reinforcement to improve reference‑image recognition while preserving video perception. The training mixes equal amounts of multi‑reference data and generic video‑caption data, preventing the model from degenerating into a simple “tag‑insertion” system.

Fact description reward : removes image tags, then checks six dimensions (subject, appearance, action, background, camera motion, style) against a video‑QA bank to ensure factual completeness.

Correct binding reward : only reference images that are both selected and placed after the correct local phrase receive positive signal; adding extra tags yields no benefit.

Distractor‑Aware Evidence Suppression (DAES) : random irrelevant images are injected; any cited image without a matching video segment incurs a penalty.

Cross‑Reference Semantic Coherence (CRSC) : for multiple views of the same entity, the evaluator extracts an “event‑entity‑reference‑group” relation and verifies that all views are aggregated under a single local description.

MRVBench Dataset

To evaluate both caption quality and grounding, the authors construct MRVBench, which contains 20,000 videos with 171,354 reference images for training and a held‑out test set of 462 videos (185 AIGC, 277 real) with 3,831 candidate images. Each sample may include up to 22 references; ~60% involve multiple images mapping to the same entity, and ~40% contain injected distractors. The benchmark splits evaluation into five aspects: factual coverage, reference selection, local phrase binding, distractor robustness, and multi‑view consistency.

Experimental Results

RefCaptioner (8B parameters) outperforms all open‑source baselines on every multi‑reference grounding metric and approaches the performance of closed‑source Gemini‑3.1‑Pro, surpassing GPT‑5.4. It also achieves the highest accuracy on the fine‑grained Video‑Caption (VDC) and VCapsBench benchmarks, demonstrating that multi‑reference training does not sacrifice ordinary captioning ability.

Ablation Study

Removing the dual‑reward (HCD‑GRPO) degrades factual description and increases false references. Adding each component (fact reward, DAES, CRSC) incrementally improves recall, grounding precision, and distractor suppression, confirming their complementary roles.

Robustness Across Reference Count

Samples are grouped by reference‑image count (2‑4, 5‑8, 9‑12, 13+). RefCaptioner maintains the highest combined score (recall × binding accuracy × distractor exclusion) in all groups, even in the hardest 13+ scenario where it still outperforms GPT‑5.4.

Video Reconstruction Evaluation

To test downstream utility, the authors reconstruct videos using the generated caption plus its selected reference images as prompts for the open‑source video generator Wan2.1‑VACE‑14B. Three multimodal experts performed pairwise Good/Same/Bad (GSB) judgments. RefCaptioner consistently achieved a higher “Good” proportion than all baselines, indicating that its grounded captions preserve more subject, appearance, action, and scene information.

Conclusion

RefCaptioner does not merely append image tags after a caption; it answers three questions during generation: which images truly relate to the video, which local visual fact each image supports, and whether irrelevant images should be rejected. Together with MRVBench, it establishes a solid foundation for accurate, controllable multi‑reference video understanding.