VisionReasoner: RL‑Unified System Beats YOLO‑World on Detection, Segmentation, Counting

Introduction

Large visual‑language models (LVLMs) have shown strong capabilities in visual dialogue, but most existing approaches rely on task‑specific modules and reward functions, limiting scalability and generalization across diverse visual perception tasks such as detection, segmentation, and counting. Inspired by test‑time reasoning advances in large language models, the authors analyze a broad set of visual tasks and observe that they can be reduced to three fundamental problem types.

Core Innovation: VisionReasoner Unified Framework

The paper proposes VisionReasoner , the first framework that integrates detection, segmentation, and counting into a shared architecture. The model consists of a reasoning module that processes the image and a segmentation module that generates masks when needed. Given an image I and a textual query T, VisionReasoner produces a structured output ({B_i, M_i})_{i=1}^{N}=F(I,T), where B_i denotes a bounding box and M_i a binary mask.

Multi‑Target Cognitive Learning Strategy

Efficient multi‑target matching using batch computation combined with the Hungarian algorithm, achieving a speedup of 6×10³⁵ over brute‑force matching.

Automatic construction of multi‑target data from raw mask annotations: bounding boxes are derived from extreme pixel coordinates and centers are computed directly from masks.

Training data are assembled from LVIS, RefCOCOg, gRefCOCO, and LISA++ datasets, yielding roughly 7,000 diverse samples.

Reward Function Design

Two families of rewards guide the reinforcement‑learning (RL) optimization:

Format Reward : penalizes repeated reasoning steps and enforces a coherent thought process.

Accuracy Reward : combines IoU‑based and L1‑based rewards for bounding boxes, masks, and key points. For example, the IoU reward assigns max{N,K} when the IoU of a matched pair exceeds 0.5; the L1 reward grants the same when the L1 distance is below 10 px for boxes or 30 px for points.

Training employs the GRPO algorithm; the policy generates multiple response samples per input, which are evaluated by the reward functions, and KL‑divergence regularization keeps the policy close to a reference model.

Experimental Evaluation

The authors evaluate VisionReasoner on ten benchmarks covering the three basic task types, including COCO (detection), ReasonSeg (segmentation), and CountBench (counting). All datasets are reformatted into a unified multimodal dialogue format and stripped of leakage information.

Key Results

Detection: +29.1% AP over Qwen2.5VL on COCO‑val.

Segmentation: +22.1% gIoU improvement on ReasonSeg.

Counting: +15.3% accuracy gain on CountBench.

Zero‑shot transfer to ten visual tasks (66,023 test samples) without task‑specific fine‑tuning.

Comparable VQA performance to state‑of‑the‑art models despite no VQA training.

Ablation Studies

Several ablations validate design choices:

Removing the non‑repetitive reward leads to longer, redundant reasoning traces and lower performance.

Varying the number of training samples shows that over‑sampling degrades generalization, confirming the importance of balanced data distribution.

Replacing the Hungarian matcher with naive pairing increases matching time from seconds to years for a 30‑object scene, confirming the efficiency of the proposed matcher.

Training on individual data sources (LVIS, RefCOCOg, gRefCOCO, LISA++) demonstrates that each contributes positively to overall performance.

Limitations

Training data are limited to 7,000 samples, which may restrict modeling of highly complex scenes.

On COCO detection, VisionReasoner still trails specialized detectors such as GroundingDINO.

Excessive sampling can cause over‑fitting, reducing generalization.

Current reasoning mechanisms struggle with tasks requiring deep physical inference or cross‑modal logical deduction.

Conclusion

VisionReasoner demonstrates that a reinforcement‑learning‑driven unified framework can effectively handle multiple visual perception tasks with a single model, achieving state‑of‑the‑art results while using modest training data. The work provides concrete insights into reward design, efficient multi‑target matching, and the benefits of structured reasoning for improving both accuracy and generalization.