How to Boost Reward Model Performance in RLHF: Data and Algorithm Strategies from the MOSS Report

Introduction

In reinforcement learning from human feedback (RLHF), the performance of the reward model (RM) is often limited by two fundamental problems: low data quality and poor generalization ability.

https://arxiv.org/abs/2401.06080
https://github.com/OpenLMLab/MOSS-RLHF

Core Issues

Low data quality : Errors and ambiguous preference labels introduce noisy training signals, preventing the RM from accurately capturing human preferences.

Poor generalization : An RM trained on a fixed distribution struggles to maintain performance on out‑of‑distribution examples, which hampers iterative RLHF pipelines.

Data‑Centric Method: Measuring Preference Strength

The authors propose a multi‑model voting scheme to quantify the strength of each preference pair.

Train N reward models on the same preference dataset, varying random seeds or initializations.

For each pair, each model outputs scores r_i^A and r_i^B. Compute the pairwise preference strength s_i = r_i^A - r_i^B.

Aggregate s_i across the N models to obtain the mean μ_i and standard deviation σ_i for that pair.

Interpretation: a mean near zero indicates noisy or incorrect labels; a large variance signals weak discriminability.

Based on the mean strength, the dataset is divided into three strata:

Low‑strength (bottom 20 %) : μ_i often negative, suggesting mislabeled preferences. Flipping the label improves downstream performance.

Medium‑strength (20 %–40 %) : μ_i close to zero, indicating ambiguous preferences. Applying soft labels (using s_i as a regression target) together with an adaptive margin loss mitigates over‑fitting.

High‑strength (top 60 %) : Strong, consistent preferences. The combination of soft labeling and adaptive margins yields the best results; however, using only the top 10 % can cause over‑fitting.

Algorithmic Enhancements

Contrastive Learning

Two contrastive schemes are explored:

Direct contrastive learning on raw preference pairs.

Contrastive learning on preference differences ( s_i), treating the difference as a metric.

Implementations using SwAV and SimCSE demonstrate notable gains in RM accuracy and downstream PPO performance.

Meta Reward Model (MetaRM)

MetaRM employs meta‑learning to align the RM with distribution shifts while preserving alignment with original human preferences. The training loop consists of six steps:

Sample a batch of preference pairs from the original dataset.

Sample a batch from a meta‑dataset that reflects a new distribution.

Compute a divergence loss on the meta‑batch (e.g., KL or contrastive loss) to encourage separation of shifted responses.

Update RM parameters θ using the gradient of the divergence loss (meta‑gradient).

Compute the standard preference loss on the original batch.

Update θ_t with the gradient of the original loss.

The combined objective maximizes the divergence loss while minimizing the original loss, enabling the RM to remain robust across distribution changes.

Experimental results show that MetaRM outperforms baseline RMs on both in‑distribution and out‑of‑distribution tasks, achieving higher PPO scores after multiple RLHF rounds without costly re‑labeling of new queries.

Conclusion

The report presents a comprehensive toolkit for improving reward‑model quality in RLHF: multi‑model preference‑strength measurement, data‑level interventions (label reversal, soft labeling, adaptive margins), and algorithmic upgrades (contrastive learning, MetaRM). Together these methods enhance both the accuracy and generalization capability of reward models, facilitating more stable and efficient RLHF pipelines.