NV-CoT: Continuous Coordinates Boost Visual Chain-of-Thought (NeurIPS 2026 Oral)

Introduction

Researchers from Nanyang Technological University, University of Science and Technology of China, National University of Singapore, and Microsoft Research propose NV-CoT (Numerical Visual Chain-of-Thought) , accepted as a NeurIPS 2026 Oral (Top 0.4%) . NV-CoT enables multimodal large models to predict continuous numerical coordinates for region selection instead of generating discrete text tokens, improving both supervised fine-tuning (SFT) and reinforcement learning (RL) pipelines with minimal architectural changes.

Background and Challenge

Existing visual chain-of-thought methods output bounding box coordinates as text tokens before invoking a cropping tool. This introduces two problems:

Training objective mismatch: Token-level cross-entropy does not penalize spatial errors proportionally (e.g., predicting 3.2 vs. 4.1 when ground truth is 3.1).

Token fragmentation: Numbers split into multiple sub-tokens scatter the complete positional meaning across the sequence, making precise generation harder.

Patch-based alternatives (e.g., PaDT) restrict region selection to a fixed grid, limiting flexibility and often requiring larger architectural modifications. NV-CoT retains the existing crop-and-zoom workflow while directly regressing continuous coordinates.

Contribution 1: Numerical Regression for Precise Localization (SFT)

NV-CoT adds four lightweight linear output heads on top of the language model to predict the top-left and bottom-right corners of a bounding box. Text generation continues via the original vocabulary; coordinates are produced by the numerical branch and passed to the cropping tool.

During SFT, the joint loss combines text cross-entropy ℒ_text with an L1 regression term on the four coordinates:

ℒ = ℒ_text + λ * Σ_{t∈T_a} |μ_t - b_t*|_1

where μ_t is the predicted box, b_t* the annotated box, T_a the coordinate action positions, and λ the regression weight. This directly reflects spatial deviation in the training objective.

On the Vis-CoT-363K validation set, localization IoU rises from 47.3 (Vis-CoT) to 59.5 (+12.2 percentage points) , providing direct evidence for the benefit of continuous coordinates.

Contribution 2: Continuous Actions for Reinforcement Learning

When intermediate box annotations are unavailable, the model must explore regions and learn from final answer rewards. NV-CoT models the bounding box as a four-dimensional independent Laplace distribution with a shared scale parameter , adding a fifth output head to predict the scale α > 0 controlling exploration magnitude.

The probability density for a box b given center μ and scale α is:

π(b|μ,α) = (1/(2α))^4 * exp(-||b-μ||_1 / α)

Training uses reparameterized sampling: b = μ + α * s ⊙ ε, where s are independent random signs and ε are independent unit-rate exponential variables. At inference, sampling is disabled and the center μ is used directly.

Integrating with GRPO , the importance ratio for a sampled box action combines the discrete token probabilities for text with the Laplace density for coordinates:

ρ = (π_new(text) / π_old(text)) * (π_new(box) / π_old(box))^4

The fourth power accounts for the four coordinates. Policy updates use group-relative advantages with rewards covering answer correctness, format compliance, and conditional tool usage — without requiring intermediate box annotations .

Experimental Highlights

Evaluated on V* Bench , HR-Bench 4K , and HR-Bench 8K (attribute, spatial reasoning, and fine-grained high-resolution perception). NV-CoT adopts each base framework's training data and configuration to isolate the coordinate modeling gain.

Both training paths improve: SFT over Vis-CoT yields +3.7, +2.0, +1.6 pp overall; RL (single-zoom) over DeepEyes yields +2.6, +1.3, +1.6 pp.

Multi-zoom retains gains: Under identical multi-call settings, NV-CoT adds +1.5, +0.8, +1.1 pp over DeepEyes.

Transfer to other tasks: On MMMU-Pro, ChartQA, and OCRBenchv2, NV-CoT improves by +2.1, +2.6, +1.5 points respectively, with gains also in object localization and counting.

Qualitative examples show clearer localization: for a handbag color question, the baseline boxes a stone wall and ground while NV-CoT targets the handbag; on a receipt QA, accurate cropping reads the correct $485 instead of the nearby $685.

Conclusion

NV-CoT demonstrates that "where to look" in visual chain-of-thought can be learned directly as continuous numerical actions , serving both SFT and RL with only a few extra linear layers. It primarily addresses cases where the model already grasps the task semantics but suffers from imprecise region localization; it cannot compensate for fundamental semantic misunderstandings.

References

Paper: https://arxiv.org/abs/2602.23959v2 — Thinking with Images as Continuous Actions: Numerical Visual Chain-of-Thought

Project page: https://kesenzhao.github.io/NV-CoT/ Code repository: https://github.com/kesenzhao/NV-CoT Hugging Face model:

https://huggingface.co/kesenZhaoNTU/NVCoT-Qwen3-VL-8B-Joint-GRPO-Step32

— NVCoT-Qwen3-VL-8B-Joint-GRPO-Step32

Key Figures