VisionReasoner: RL‑Unified Model Beats YOLO‑World Detection, Segmentation, Counting

Introduction

Large visual‑language models (LVLMs) have shown strong abilities in visual dialogue, yet they rely on task‑specific modules and reward functions, limiting scalability and generalisation across diverse visual perception tasks such as detection, segmentation, and counting. Inspired by test‑time reasoning in large language models, the authors analyse these tasks and observe that they share a common multi‑target structure.

Core Innovation

The paper introduces VisionReasoner , the first unified framework that integrates detection, segmentation, and counting within a single shared architecture. The key contributions are:

Design of a multi‑target cognition learning strategy that includes (a) an efficient Hungarian‑algorithm‑based matching mechanism for batch processing, and (b) an automatic construction of multi‑target data from raw mask annotations (bounding boxes and centre points).

Construction of a composite reward function comprising format rewards (thought‑process constraints, non‑repetition penalties) and accuracy rewards (IoU/L1 joint optimisation, multi‑target matching maximisation).

Development of a structured reasoning generation module that produces interpretable intermediate steps, improving task generalisation.

Methodology

Task Reformulation

All visual perception tasks are reformulated into three basic types:

Detection: generate a set of bounding boxes for queried objects.

Segmentation: generate binary masks for queried regions.

Counting: estimate the number of queried objects.

Each task is expressed as a unified input‑output pair ( { B_i, M_i } )_{i=1}^{N} = F( I, T ), where B_i denotes a bounding box and M_i a mask.

Model Architecture

VisionReasoner builds on the Seg‑Zero backbone, adding a multi‑target cognition head. The model first processes the image to locate objects, then optionally generates masks, and finally outputs a structured answer consisting of bounding boxes, centre points, and masks as required by the task.

Reward Functions

The reward suite contains:

Format Reward : encourages the model to emit a coherent reasoning chain and penalises repeated inference patterns.

Accuracy Reward : includes Bboxes IoU Reward (max‑pairwise IoU > 0.5 yields max{N,K}), Bboxes L1 Reward (L1 distance < 10 px yields max{N,K}), and analogous rewards for centre‑point L1 distances (< 30 px threshold).

During training, the policy model generates multiple response samples per input; each sample is scored by the composite reward and optimised via KL‑regularised reinforcement learning (GRPO algorithm).

Efficient Multi‑Target Matching

To compute rewards for many‑to‑many predictions, the authors batch‑process all predicted and ground‑truth boxes/points and apply the Hungarian algorithm for optimal one‑to‑one assignment. This yields a speed‑up of roughly 6×10^{35} over naïve brute‑force matching.

Experiments

Evaluation Benchmarks

Ten diverse benchmarks covering the three task types are used: COCO and RefCOCOg for detection, ReasonSeg and LISA for segmentation, PixMo‑Count and CountBench for counting. All datasets are normalised to a unified cross‑modal dialogue format, and standard metrics (COCO AP, gIoU, counting accuracy) are reported.

Results

Despite training on only 7,000 samples, VisionReasoner‑7B achieves:

+29.1 % AP improvement over Qwen2.5VL on COCO detection.

+22.1 % gIoU gain on ReasonSeg segmentation.

+15.3 % counting accuracy increase on CountBench.

Compared with specialised models (e.g., YOLO‑World, GroundingDINO), VisionReasoner remains competitive, especially in zero‑shot transfer across ten tasks (66,023 test samples).

Ablation Studies

Inference Length : longer reasoning chains are generated for complex queries (ReasonSeg) and shorter ones for simple category names (COCO).

Multi‑Target Matching : the Hungarian‑plus‑batch approach reduces matching time from years (brute force) to seconds, a speed‑up of several orders of magnitude.

Non‑Repetition Reward : models trained with this reward produce shorter, less redundant reasoning and achieve higher performance.

Training Data Diversity : adding LVIS, RefCOCOg, gRefCOCO, and LISA++ progressively improves results, confirming the benefit of varied textual annotations.

Sampling Quantity : performance rises with more samples up to a point, after which over‑sampling harms generalisation.

Qualitative Findings

Visualisations show that VisionReasoner can simultaneously output detection boxes, segmentation masks, and counts while providing an interpretable reasoning trace. The model distinguishes similar objects and adapts reasoning length to query complexity.

Limitations

Training data size (7 k samples) caps modelling of highly complex scenes.

Still lags behind some task‑specific models (e.g., GroundingDINO) on raw detection AP.

Over‑sampling can degrade generalisation, requiring careful balance.

Current inference mode struggles with deep physical reasoning or cross‑modal logical deduction.

Conclusion

VisionReasoner demonstrates that a reinforcement‑learning‑driven unified framework can effectively handle multiple visual perception tasks, achieving state‑of‑the‑art results with modest data and offering interpretable, structured reasoning. The work provides key insights for extending RL techniques to large visual‑language models.