How Visual Para-Thinker Tackles Visual Hallucination with a Clever Parallel Reasoning Design

Current visual reasoning research often extends inference length, but vertical expansion leads to exploration rigidity and attention drift, causing severe visual hallucinations. Existing models such as K2.5, Step3‑VL, and LongCat‑Flash‑Thinking explore width expansion, yet challenges remain.

To address this, the authors propose Visual Para-Thinker , the first parallel‑thinking framework designed for large vision‑language models. The framework integrates Pa‑Attention (parallel attention) and LPRoPE (Learnable Parallel Rotary Position Embedding) to achieve isolation, unbiasedness, and distinguishability of parallel inference paths.

The visual token space is divided using two strategies:

Block partition : assigns distinct image quadrants (e.g., top‑left, bottom‑right) to separate paths, producing unique attention distributions per region.

Scan partition : defines predefined scanning orders (left‑to‑right, top‑to‑bottom, etc.) so each path follows a different visual scan trajectory.

Block partition can cause redundant computation across paths, while scan partition may reduce path diversity. To combine their strengths, a hybrid training strategy mixes data generated from both partitions.

Isolation is enforced via Path‑aware Attention , which uses special <think i> tokens to keep contexts of different paths separate, unlike causal attention.

Unbiasedness is achieved by assigning the same position‑id range to the start token of every path during the parallel stage, avoiding the positional ordering bias that arises when different paths receive distinct position‑id intervals.

Distinguishability is restored by introducing LPRoPE , which adds a learnable path‑specific absolute position embedding before applying rotary position encoding, ensuring that the model can differentiate between paths despite shared position‑id ranges.

For training, a parallel inference dataset of 163,000 question‑answer pairs is built from LVIS, LAION, COCO, PixMoCount, RefCOCO, RefCOCO+, and RefCOCOg. The teacher model Qwen3‑VL‑235B‑A22BInstruct (temperature 0.1) generates four visual‑centric paths per sample using a hybrid block‑and‑scan strategy, while higher‑temperature Qwen3‑VL‑30B‑A3B and InternVL3 5‑241B‑A28B provide diverse data and verification.

Experimental evaluation on visual perception tasks (Pixmo, CountBench, V*, MMVP, HallusionBench, RefCOCO) shows consistent gains: on V* tasks, the 3B and 7B variants improve by 12.6 and 6.3 points respectively; on HallusionBench, improvements of 6.1 and 5.0 points are observed; grounding tasks also see modest gains over the original Qwen2.5‑VL.

Further analysis reveals task‑dependent partition preferences: counting tasks benefit from scan partition to avoid overlapping region bias that can induce hallucination, whereas block partition offers explicit attention allocation useful for other scenarios.

In summary, Visual Para-Thinker demonstrates that parallel visual reasoning with carefully designed attention and positional mechanisms can alleviate visual hallucination and enhance multimodal performance, and the authors anticipate extending parallel‑thinking techniques such as RL‑based multi‑round reasoning and agentic RL to this framework.