RL-Driven Diffusion Models: From Generation to Decision-Making (Survey)

Why Diffusion Models Need Reinforcement Learning

Traditional diffusion models generate content through multi-step denoising, excelling at fitting existing data distributions but not necessarily satisfying human preferences or complex task requirements. For example, lower noise prediction error does not guarantee better aesthetic alignment or successful robot task completion. Reinforcement learning introduces reward signals to adjust the model based on actual generation outcomes, shifting from static data fitting to interactive experience, from single-path imitation to closed-loop planning with self-correction, and from minimizing MSE loss to optimizing concrete reward functions. In this view, each denoising step becomes a decision, and the full generation trajectory forms a decision sequence optimized via final rewards.

How Reinforcement Learning Enters Diffusion Models

Feedback Signals

RLHF : Uses human preference data to train a reward model, then optimizes generation policy via RL.

DPO : Directly adjusts the model using paired preference data without an explicit reward model.

RLVR : Provides optimization signals based on verifiable metrics such as answer correctness or task completion.

Policy Optimization Algorithms

PPO : Constrains policy update magnitude for training stability.

GRPO : Computes advantages by comparing multiple generations for the same input, eliminating the need for a separate value model.

Note: RLHF, DPO, and RLVR are feedback/optimization paradigms, while PPO and GRPO are policy optimization algorithms; they operate at different levels.

The core difficulty for diffusion models is computing the probability of multi-step generation trajectories and effectively back-propagating final rewards to each step.

Expanding Capabilities Across Domains

Discrete Diffusion Language Models: Stronger Reasoning

Discrete diffusion LLMs (dLLMs) iteratively recover masked tokens. Key challenge: computing generation probabilities across multiple unmasking rounds and feeding task rewards back to the entire trajectory. Surveyed approaches: d1: Introduces policy gradients into dLLMs. DiffuCoder: Uses complementary masking to reduce probability estimation noise. ESPO: Optimizes the full generation sequence as a whole. DCoLT: Treats unmasking as a latent reasoning chain, optimizing via outcome rewards. dUltra: Adds a learnable unmasking planner to improve accuracy while reducing steps.

These works show RL optimization extending from final outputs to generation order, unmasking strategies, and the reasoning process itself.

Multimodal Models: From Preference Alignment to Complex Task Control

RL has been applied to image, video, 3D, audio, and action generation, plus editing and unified understanding-generation tasks.

Image generation : Flow-GRPO brings GRPO to diffusion training. On Stable Diffusion 3.5-M, GenEval score rises from 0.63 to 0.95, OCR benchmark from 0.59 to 0.92.

Video generation : Targets visual quality, motion smoothness, flicker reduction, and subject consistency. VideoDPO uses preference data; VADER back-propagates reward gradients through video denoising; DanceGRPO extends GRPO to image and video tasks.

Further coverage includes 3D content, text-to-audio, and image editing.

Overall, RL in multimodal domains now addresses semantic alignment, structural consistency, and complex task control beyond mere quality improvement.

Robotics: From Generating Actions to Completing Tasks

Two main integration patterns:

Diffusion policy trained with RL : Diffusion model serves as policy network; RL optimizes action generation. Split into online (continuous environment interaction) and offline (pre-collected datasets) approaches.

Generative action experts in Vision-Language-Action (VLA) models : Diffusion or Flow Matching models act as action modules combined with vision-language perception. Examples: π₀ (Flow Matching action expert generating continuous actions from VLM features) and π_RL (adds online PPO to Flow-based VLA for policy optimization). Note: π₀ is a generative action expert, not an RL method itself.

World-model-assisted VLA optimization: VLA-RFT builds simulated environments via world models for verifiable reward training; World-VLA-Loop couples video world models with VLA policies for interactive closed-loop behavior adjustment.

These studies demonstrate that RL combined with generative action models enables robots to adjust strategies based on environmental feedback and accomplish real tasks, not just imitate recorded actions.

Scientific Computing: Enforcing Physical Laws

Diffusion models used for molecular design, protein structure prediction, crystal generation, and physical field modeling must satisfy energy, geometry, and PDE constraints. Surveyed works:

Biomolecular design : RL-DIF optimizes sequences with protein foldability rewards; BetterBodies optimizes for antibody binding affinity.

3D molecular structure : RLPF uses molecular force-field feedback; MDRL guides design with docking scores and LogP.

Crystal generation : MatInvent and RLFEF use formation energy as reward.

Physical field modeling : PIRF leverages PDE residuals and boundary conditions to enforce physical constraints.

Scientific imaging : ASMR applies RL to MRI k-space sampling location selection.

Key value: evaluation shifts from visual similarity or data distribution to computable scientific metrics and physical laws.

Three Outstanding Challenges

Reward design : If the reward function does not accurately reflect the task goal, models may achieve high rewards without achieving the intended effect.

Exploration-exploitation balance : Diffusion models involve many continuous or discrete decision steps; balancing exploration of new generation paths with exploitation of known experience is critical in high-dimensional spaces.

Sparse reward instability : In some RL paradigms, reward arrives only after the full generation; effectively using this sparse feedback to optimize intermediate steps remains unsolved.

Conclusion

From discrete diffusion LLMs to image, video, multimodal generation, robotics, and scientific computing, RL is pushing diffusion models from data distribution fitting toward task-oriented policy optimization. Reward feedback enables exploration of diverse generation paths and strategy adjustment based on task outcomes, extending applications beyond content generation into complex reasoning, dynamic interaction, and multi-step decision-making. However, reward reliability, exploration efficiency in high-dimensional spaces, and training stability under sparse rewards remain key hurdles for real-world deployment.