Diffusion Reward Models: Learning Human Preference Distributions Beyond Scalar Scores
This paper introduces Diffusion Reward Models (DRM) that replace the scalar value head with a Diffusion Transformer to model full reward distributions, capturing multimodal human disagreement, enabling uncertainty-aware decisions, test-time scaling via repeated reward sampling, and improving downstream RLHF performance.
