Look-Back Triggers Visual Reflection in Qwen-2.5-VL, +6.3% Perception

Problem Statement

Multimodal large language models (MLLMs) often rely heavily on textual information in the later stages of reasoning, causing visual inputs to be ignored. Existing solutions inject visual tokens explicitly, increasing inference complexity and not fully exploiting the model's inherent visual‑fusion capabilities.

Key Observation

By analyzing attention patterns, the authors discovered that, when guided with a simple prompt, MLLMs can spontaneously shift attention back to visual regions in the reasoning tail without any explicit visual injection. This suggests that MLLMs possess an implicit visual‑reflection ability.

Look-Back Method

Look-Back is an implicit training paradigm that encourages MLLMs to "look back" at visual information during inference. It consists of two stages:

Cold‑start Supervised Fine‑Tuning (SFT) : High‑quality <back> tokens are generated using a stronger model (e.g., GPT‑4o) to create a reflective reasoning dataset. Two variants are defined:

Semantic‑back : Triggers during intermediate reasoning to re‑examine crucial visual details.

Solution‑back : Triggers after an initial answer is produced, prompting a full visual re‑assessment.

Reinforcement Learning (RL) : The GRPO algorithm is employed with a formatted reward that combines accuracy and a penalty for missing <back> tokens. The reward encourages the model to generate the <back> token autonomously, thereby activating visual reflection.

Experimental Setup

Experiments were conducted on Qwen‑2.5‑VL‑7B across eight multimodal benchmarks (five math‑oriented and three perception‑oriented). Baselines included closed‑source models (GPT‑4o, o3), open‑source general MLLMs (Qwen‑2.5‑VL‑32B, InternVL3‑38B), and open‑source inference‑only MLLMs (e.g., MM‑Eureka‑8B, R1‑VL‑7B). Training used eight NVIDIA A800 GPUs, with SFT for one epoch and RL for two epochs (batch size 128, temperature 1.0, reward weight 0.1).

Results

Performance Gains : Look‑Back improved perception tasks by an average of 6.3 % (e.g., Semantic‑back from 61.3 % to 67.6 %) and math tasks by ~7 % (Semantic‑back from 48.5 % to 55.5 %).

Competitive Edge : Despite having far fewer parameters than GPT‑4o, the Solution‑back variant narrowed the gap, especially on the Solution‑back setting.

Generalization : Training primarily on math data still yielded strong gains on perception benchmarks, indicating cross‑task adaptability.

Ablation Studies

Removing either the SFT or RL stage caused a significant drop in performance, confirming the necessity of both components. Varying the reflection rate showed optimal values between 30 %–50 %; too low or too high rates degraded results.

Limitations

Cold‑start data relies on closed‑source models (GPT‑4o) to generate <back> tokens, limiting open‑source reproducibility.

Trigger rate remains modest (average ~6 % with prompt‑only guidance), requiring RL to boost activation.

Training data bias toward mathematical reasoning hampers maximal perception performance.

Potential reward‑gaming attacks where the model emits empty <back> sequences to collect format rewards.

Method sensitivity to model architecture and pre‑training; earlier Qwen‑2‑VL showed poor reflection.

Discussion

The authors analyze failed attempts (e.g., using weaker Qwen‑2‑VL, reward attacks) and propose expanding cold‑start data with more diverse modalities to improve perception generalization. They also compare semantic‑back vs. solution‑back, noting that early visual re‑focus benefits perception, while later re‑focus aids complex math reasoning.

Conclusion

Look‑Back demonstrates that MLLMs can be trained to autonomously re‑activate visual attention without extra image tokens or architectural changes, yielding consistent improvements across multimodal reasoning benchmarks.