DiffusionART: 10x Faster Video Diffusion RL via Aggregation-First Reward Tilting

Researchers from the National University of Singapore, Kling AI, and HKUST present DiffusionART (Diffusion Reinforcement with Aggregation-First Reward Tilting), a new online reinforcement learning algorithm for diffusion-based image and video generation. The method addresses fundamental instability and inefficiency issues in prior forward-process RL approaches like DiffusionNFT.

Background: Two Paradigms and Their Flaws

Diffusion post-training splits into two paradigms:

Reverse-process RL (e.g., Flow-GRPO): models the reverse denoising as an MDP, uses ODE-to-SDE conversion for policy gradients. Suffers from poor credit assignment because final reward is uniformly distributed across all denoising steps, limiting sample efficiency.

Forward-process RL (e.g., DiffusionNFT): re-noises generated outputs and performs reward-weighted regression via positive and negative branches. Avoids transition probabilities and supports efficient samplers, but stability depends on joint training of both branches and task-specific EMA off-policy schedules. Single-branch or on-policy updates often collapse.

The authors identify three unexplained phenomena in DiffusionNFT:

In theory, moving toward positive samples and away from negative samples should yield the same gradient direction. Why does removing one branch cause collapse?

Why is off-policy training required for stability? What is the essential difference between on-policy and off-policy in continuous-space diffusion RL?

Three Mechanistic Insights

Insight 1: Forward–Reverse Consistency Force

Expanding the NFT update in output space reveals, besides the reward-improving guidance, a forward–reverse consistency force arising from the discrepancy between the ideal straight-line forward velocity field (defined by linear noising) and the actual reverse velocity field used during generation. This force does not improve reward but can produce large parameter gradients, causing gradient spikes and reward collapse. The problem is not missing reward direction but an extra consistency force contaminating optimization.

Insight 2: Dual Branches Partially Cancel the Consistency Force

Positive and negative branches carry reward guidance in the same direction but consistency forces in opposite directions. Joint training partially cancels the consistency force, explaining why single-branch training collapses. However, cancellation is incomplete: reward normalization balances coefficients only within a sample group, not per state. After mapping through the model Jacobian to parameter space, significant residual gradients remain. Dual branches mitigate but cannot eliminate the consistency force by objective design.

Insight 3: EMA's Stabilizing Mechanism Is a Restoration Force, Not Just Off-Policy Data

Ablation shows that keeping EMA rollout but replacing the loss's old-policy anchor with the current policy still leads to collapse. Gradient decomposition reveals EMA introduces a restoration force pulling the current model back toward the old policy; the further the drift, the stronger the force. Even without EMA, splitting each rollout into two exclusive minibatches and updating twice consecutively generates a similar restoration force and avoids collapse in the observed interval, but at a 15× efficiency cost on an OCR task. Counter-intuitively, the restoration force's RMS in output space is 9.4× smaller than guidance, yet its aggregated parameter-space gradient norm is 4.9× larger, showing a powerful gradient transformation effect: a tiny output-space restoration force can overwhelm reward guidance, yielding stable but inefficient training.

DiffusionART Design

Two design goals emerge: (1) eliminate consistency force at the objective-construction level, not via cross-sample gradient cancellation; (2) obtain adaptive restoration without EMA tuning.

1. Aggregation-First: Aggregate Rewards, Then Guide Real Denoising States

For a given prompt, ART first computes a shared reward-improving direction from a set of generated samples and their rewards: g = Cov(reward, sample) (RMS-normalized) From a reward-tilting perspective, up-weighting high-reward samples shifts the generation distribution mean along this covariance direction. RMS normalization unifies guidance scale across reward ranges, removing the need for manual guidance-strength tuning.

This shared guidance is then applied to the real ODE rollout states via MSE regression against a frozen target: L = || v_θ(s_t) - (v_θ_old(s_t) + g) ||^2 Unlike NFT, this objective does not rely on the ideal forward velocity field, so the consistency force is eliminated by construction. This is the essence of Aggregation-First: instead of injecting per-state guidance contaminated with consistency force and hoping gradients cancel, ART first aggregates a clean reward direction at the endpoint level, then applies it to each real denoising state.

2. Two Updates: Adaptive Restoration Without EMA

ART splits each rollout's data into two exclusive minibatches, each updated once while sharing the same frozen target. The first update moves from the old policy primarily along the reward direction G . By the second update, the model has drifted, naturally producing a restoration term:

Δθ_2 ≈ -η ∇_θ v_θ(s_t)^T (v_θ(s_t) - v_θ_old(s_t))

Larger drift → stronger restoration; small drift → reward guidance dominates. This provides adaptive restoration without EMA or extra hyperparameters, and avoids re-training the same minibatch.

3. Trust-Region Mask: Limit Excessive Drift

In the second update, a PPO-style clip mask is applied: if a state's deviation from the old policy exceeds a threshold and the update would push it further outward, that update is masked; inward (restorative) updates are kept. The mask uses a fixed configuration, no per-task tuning. The entire process uses deterministic ODE sampling, avoiding ODE-to-SDE conversion and transition-likelihood computation.

The team also released DiffusionART-Flash , which randomly samples two steps from high-noise regions during training, achieving similar performance to full-trajectory training with much reduced compute. The open-source repo now includes RL recipes for frontier open video models such as Minimax-H3.

Experimental Results

Video: Same Reward, Drastically Lower GPU Hours

On HunyuanVideo-13B with combined visual-quality (VQ) and motion-quality (MQ) rewards, targeting each baseline's late-stage average reward:

ART reaches DiffusionNFT's late average reward 12.0× faster .

ART reaches Flow-GRPO-Fast's late average reward 11.7× faster .

ART's own late average reward improves 28.8% (VQ) and 53.0% (MQ) over the baselines' late averages.

On Wan2.1-I2V-14B (image-to-video), reaching MQ=6:

ART: 45.8 GPU hours

Flow-GRPO-Fast: 220.3 GPU hours

Efficiency gain: 4.8× . Other algorithms plateau quickly; ART continues stable improvement.

Image: Matches Tuned NFT Efficiency Without Per-Task EMA Schedules

On SD3.5-M , tested on GenEval, OCR, PickScore, and joint optimization of GenEval, PickScore, CLIPScore, HPSv2.1. ART's training efficiency matches NFT with the original paper's per-task EMA settings, while avoiding the severe collapses of on-policy NFT. To reach Flow-GRPO's step-500 reward, ART requires 10.0× (GenEval), 5.2× (OCR), 9.6× (PickScore) fewer steps.

The same core method scales to FLUX.1-dev (12B) , confirming applicability beyond a single small image model to larger image models and higher-dimensional video tasks.

Ablation (Fig. 7 right) confirms Guidance and Restoration are both necessary: covariance guidance alone with a single update gives fast early OCR gains but later degradation; adding the second update's restoration enables long-term stability. Even with tuned EMA (β=0.1) and two updates, NFT still collapses, proving stability comes from clean reward guidance plus adaptive restoration, not merely from updating twice.

Pairwise Rewards and Small Groups: Cutting Expensive Video Rollouts

DiffusionART natively supports pairwise reward models, converting preferences to win/loss labels and reusing the same covariance guidance and MSE update, without differentiable reward models. In VQ-only pairwise video experiments with K=2 videos per prompt (vs. K=8), under the same 400-update budget:

K=2: 260.8 GPU hours , final VQ = 12.235

K=8: 883.3 GPU hours , final VQ = 12.092

Compute reduction: ~3.4× with comparable final quality.

This shows ART does not require large group sizes for effective reward improvement, critical for video RL where rollout cost is extreme.

Qualitative Examples

The article includes generated image and video samples demonstrating improved visual quality and motion coherence, such as "three suitcases" and "teddy bear with car in jungle" images, and a "waterfall flowing into emerald pool in tropical rainforest" video.

Conclusion

DiffusionART's value goes beyond faster curves. By decomposing gradients, it answers a foundational question: when fine-tuning a diffusion model with reward, what forces act on the model, how large are their gradients, and how do they interact? The analysis separates reward guidance, hidden consistency force, and off-policy restoration. The resulting method removes consistency force at the objective level and supplies adaptive restoration via a simple two-update scheme, rather than suppressing all instabilities with stronger EMA. This yields deeper insight and a cleaner design recipe for diffusion post-training: efficiency and stability can stem from understanding the objective, not just task-specific hyperparameter tuning.

Paper & Author Information

Paper : DiffusionART: Efficient, Stable and Scalable Reinforcement of Diffusion Models

Authors : Zhenglin Wan, Jiayu Chen, Zhengfeiyang Zhang, Pengfei Zhou, Wangbo Zhao, Fangyuan Kong, Pengfei Wan, Kun Gai, Yang You

Affiliations : National University of Singapore; KlingAI Research; HKUST

Contacts : [email protected], [email protected]

Links : Paper (arXiv:2610.12572), Code (github.com/vanzll/DiffusionART), Hugging Face (huggingface.co/collections/vanzl/diffusionart), Project (vanzll.github.io/diffusionart-project/)