What’s Next for Visual Reinforcement Learning? A Comprehensive 2024‑2025 Survey

1 Introduction

Reinforcement learning (RL) has achieved remarkable success in large language models (LLMs), especially through paradigms such as Reinforcement Learning from Human Feedback (RLHF) and recent frameworks like DeepSeek‑R1, which improve generation quality and enable nuanced reasoning.

Inspired by these advances, the community has rapidly extended RL techniques to multimodal large models, including vision‑language models (VLMs), vision‑language‑action (VLA) agents, diffusion‑based visual generation, and unified multimodal frameworks. Notable examples such as Gemini 2.5 use RL to align visual‑text reasoning, while VLA models apply RL for complex sequential decision‑making in GUI automation, robotic manipulation, and embodied navigation.

The surge of diffusion‑based generative models has further driven RL‑driven innovation. Methods like ImageReward incorporate RL into the diffusion process to improve semantic consistency and visual fidelity via human‑preference or automated reward feedback. Unified models that combine understanding and generation increasingly rely on RL‑fine‑tuning to achieve better generalization and task transfer.

Despite these breakthroughs, core challenges remain: stabilizing policy optimization under complex reward signals, handling high‑dimensional and diverse visual inputs, and designing scalable reward functions that support long‑horizon decision making.

Survey Scope and Taxonomy

This survey systematically summarizes the latest progress (since 2024) in visual reinforcement learning, reviewing more than 200 representative papers. We first revisit RL successes that laid the foundation for multimodal adaptation, then trace their evolution in the visual domain. The works are grouped into four major pillars:

Multimodal large language models

Visual generation

Unified RL frameworks

Vision‑language‑action agents

For each pillar we analyze algorithmic design, reward modeling, and benchmark advancements, and we extract emerging trends such as curriculum‑driven training, preference‑aligned diffusion, and unified reward modeling.

Key Contributions

Systematic and Up‑to‑Date Review : A comprehensive catalog of over 200 visual RL studies covering multimodal LLMs, visual generation, unified models, and VLA agents.

Technical Analysis : Detailed examination of progress in policy optimization, reward engineering, and evaluation protocols, highlighting challenges in reward design for generation and the lack of intermediate supervision in VLA tasks.

Methodological Framework : A taxonomy that classifies visual RL methods by metric granularity and reward supervision, outlining three image‑generation reward paradigms and providing actionable guidance for selecting and developing RL strategies.

We also review evaluation protocols that address set‑level fidelity, sample‑level preference, and state‑level stability, and we point out open challenges such as sample efficiency, generalization, and safe deployment.

Resources and references, including the curated GitHub repository

https://github.com/weijiawu/Awesome-Visual-Reinforcement-Learning

and the arXiv preprint https://arxiv.org/abs/2508.08189, are provided for further exploration.