The First Visual‑Language Parallel Thinking Framework: Unpacking Its Core Mechanisms

Motivation

Current test‑time expansion paradigms mainly increase reasoning length, but vertical expansion often leads to exploration rigidity. Recent models such as K2.5, Step3‑VL and LongCat‑Flash‑Thinking have begun to explore width expansion. In visual tasks, longer reasoning sequences cause attention drift and severe visual hallucination.

Visual Para‑Thinker

We propose Visual Para‑Thinker , the first parallel‑thinking framework designed for large‑scale visual‑language models. By integrating Pa‑Attention (parallel attention) and LPRoPE (Learnable Parallel Rotary Position Embedding), the framework achieves three properties for parallel reasoning paths: isolation, unbiasedness, and distinguishability.

Parallel Reasoning Paths: Visual‑Centric Partitioning

We define two visual‑centric partition modes:

Block partition : each path attends to a specific image sub‑region (e.g., top‑left, top‑right, bottom‑left, bottom‑right), producing distinct attention distributions.

Scan partition : each path follows a predefined scanning order (left‑to‑right, top‑to‑bottom, right‑to‑left, bottom‑to‑top), yielding unique attention sequences.

Block partition offers explicit regional attention but may cause redundant computation across paths; scan partition is computationally simple but can reduce path diversity. We therefore adopt a hybrid training strategy that mixes data generated by both partitions.

Framework Stages

Parallel thinking stage : using the shared context, visual partitioning assigns distinct reasoning directions to each path.

Summarization stage : background information from all parallel paths is aggregated to produce the final answer.

Isolation

We introduce Path‑aware Attention , which inserts a special <think i> token for each path, ensuring that attention computation remains isolated between paths, unlike causal attention.

Unbiasedness

Previous methods gave each path a separate position‑id range, creating inherent ordering bias (e.g., “loss in the middle”). Instead, we assign the same position‑id range to all paths; the start token of each path shares the same position id, while the summarization token uses the position id of the longest path plus one, eliminating positional bias.

Distinguishability

Sharing position ids harms the ability to differentiate paths. To restore distinguishability, we propose LPRoPE : before applying rotary position encoding, we add a learnable absolute position embedding specific to each path, then combine it with rotary encoding, allowing the model to tell paths apart.

Data and Training

We construct a parallel‑reasoning dataset of 163,000 question‑answer pairs sourced from LVIS, LAION, Microsoft COCO, PixMoCount, RefCOCO, RefCOCO+ and RefCOCOg. The teacher model Qwen3‑VL‑235B‑A22BInstruct generates four visual‑centered reasoning paths per sample using a hybrid of block and scan partitions. Additional diversity is introduced with high‑temperature outputs from Qwen3‑VL‑30B‑A3B‑Instruct and InternVL3 5‑241B‑A28B.

Experiments

We evaluate on visual perception tasks: counting (PixMo, CountBench), visual search (V*), hallucination (MMVP, HallusionBench), and grounding (RefCOCO). Results show consistent improvements:

V* task: +12.6 (3B) and +6.3 (7B) points.

HallusionBench: +6.1 (3B) and +5.0 (7B) points.

Grounding tasks also gain over the baseline Qwen2.5‑VL.

Further analysis reveals task‑dependent preferences for partition modes. Counting tasks benefit from scan partition because block partition can cause overlapping region bias and hallucination, whereas block partition provides explicit regional attention useful for other tasks.

Conclusion and Future Work

Visual Para‑Thinker demonstrates that parallel‑thinking frameworks can substantially boost visual‑language model performance. Future directions include integrating parallel‑thinking with reinforcement learning, multi‑round reasoning, and Agentic RL to achieve faster and larger‑scale extensions. As more base models (e.g., K2.5, Step3‑VL, LongCat‑Flash‑Thinking) adopt parallel thinking, we anticipate the paradigm will unlock significant potential.