Vision‑Reasoning Model: Enabling LLMs to See and Think

DeepSeek’s DeepSeek‑R1 achieved strong reasoning performance but cannot accept image inputs. Its product line includes DeepSeek‑V3, a visual‑language model (VLM) that supports image and text like GPT‑4o, and DeepSeek‑R1 (a large reasoning model, LRM) that only processes text yet incorporates a chain‑of‑thought reasoning process similar to OpenAI‑o1.

Why a Vision‑Reasoning Model?

The current VLMs suffer from weak reasoning abilities, while LRMs lack visual perception. For example, a student may ask a physics question that includes an image; the ideal model must both understand the visual content and perform a reasoning process before producing the final answer.

Desired Capabilities

Interpret image content.

Execute a reasoning loop (think, re‑evaluate, consider alternatives) before outputting the answer.

VLM Architecture (LLaVA Example)

LLaVA uses a CLIP visual encoder to convert an image into a vector. A trainable linear layer with weight matrix W projects this vector to the same dimension as the language tokens. The projected visual hidden state H_v is concatenated with the textual token hidden states and fed into a Vicuna LLM (based on a Transformer). The final prediction is conditioned on both image and text information.

Training the VLM

Training is performed via end‑to‑end fine‑tuning. The CLIP visual encoder remains frozen; only the linear projection layer W and the Vicuna LLM parameters (denoted ϕ) are updated.

Reinforcement‑Learning Fine‑Tuning for Visual Reasoning

To endow the model with reasoning, the article defines an image‑classification task where each sample contains an image, a caption, and a question whose answer is the caption. Two reward types are introduced:

Correctness reward : +1 when the model outputs the correct class (e.g., "dog").

Format reward : granted when the model wraps its reasoning inside <think>…</think> tags and places the final answer inside <answer>…</answer> tags.

This reward scheme forces the model to generate explicit reasoning steps before answering, a technique shown to improve LLM reasoning in text‑only settings and expected to benefit visual reasoning as well.

Application Scenario

An example shows GPT‑4o misreading a medical‑monitor graph and outputting an incorrect value, illustrating the need for a model that can both see the image and reason about it.

Conclusion

The article argues that a Large Vision Reasoning Model (LVRM) that integrates visual perception with chain‑of‑thought reasoning will be the next major breakthrough. It has detailed the VLM construction, the fine‑tuning process, and a reinforcement‑learning framework that together lay the groundwork for such models.