A New Paradigm for LLM Reward Modeling: Mixing Huber and Hinge Losses in E‑GRM

1. Background and Motivation

Traditional reward‑model training for LLMs faces a dilemma: mean‑squared error (MSE) provides precise score fitting but is sensitive to outliers, while Bradley‑Terry‑style contrast losses yield good ordering but lack absolute score calibration. The E‑GRM framework requires the scorer to produce continuously comparable scores (regression) and to supply stable contrast signals (ranking) for downstream GRPO optimization, motivating a loss that satisfies both goals.

2. E‑GRM Mixed‑Loss Methodology

2.1 Overall Scorer Objective

The scorer is trained to minimize a weighted sum of two components, where ŷ denotes the model output and y the human‑annotated target. The weighting hyper‑parameter emphasizes regression as the primary objective and ranking as a secondary constraint.

2.2 Huber Loss: Balancing Fine‑Tuning and Noise Robustness

Huber loss uses a piecewise definition: when the prediction error is within a threshold δ (default value omitted), it behaves as L2 loss, providing dense gradients for fine‑tuning; when the error exceeds the threshold, it switches to L1 loss, capping gradient magnitude to prevent a few noisy labels from dominating optimization.

2.3 Hinge Loss: Focusing on Hard Examples

The hinge component contributes gradients only when the score gap between a positive and a negative sample is insufficient. With a margin m, loss is zero once the positive score exceeds the negative by more than m, allowing the optimizer to ignore already‑correct pairs and concentrate on borderline or mis‑ranked cases.

2.4 Training Dynamics of the Combined Loss

Objective Complementarity : Huber drives "score fitting" while hinge drives "order correctness".

Gradient Complementarity : Huber supplies dense gradients to avoid under‑fitting; hinge supplies sparse gradients to curb over‑fitting.

Noise Robustness : Huber’s linear region mitigates outlier influence; hinge’s margin filters noisy negative examples.

2.5 Integration with the E‑GRM Pipeline

After training, the scorer is embedded in the long‑chain reasoning path as a quality evaluator; its outputs also serve as the contrast reward in the extended GRPO stage, making the mixed‑loss scorer the central link of the "dynamic trigger → scoring → policy optimization" loop.

3. Experimental Evaluation

3.1 Loss Ablation on RM‑Bench

Loss Config   RM‑Bench   RMB Overall   Spearman
Only MSE      73.4%      0.687         0.612
Only Hinge    71.8%      0.671         0.583
Only Huber    75.1%      0.702         0.654
Mixed (α=0.7) 79.2%      0.743         0.731

The mixed loss improves all three metrics, confirming a synergistic benefit rather than a simple additive effect.

3.2 Sensitivity to the Weight α

Varying α shows that a moderate value (≈0.7) is optimal; too small (hinge‑dominant) degrades calibration, while too large (Huber‑dominant) reduces hard‑example discrimination.

3.3 Impact of the Huber Threshold δ

A very small δ pushes most samples into the L1 region, losing fine‑grained gradients; a very large δ eliminates noise robustness. The chosen δ yields the most stable performance on the E‑GRM training set.

3.4 Hard‑Example Focusing

During later training stages, the proportion of samples activating the hinge loss drops from 78% to 12%, indicating that the scorer quickly resolves easy cases and continues learning only on truly difficult examples—behavior absent in a pure MSE loss.

3.5 Downstream GRPO Performance

When the mixed‑loss scorer provides the GRPO reward, the policy model converges about 30% faster and achieves a final alignment score 2.1% higher than when using an MSE‑only scorer, demonstrating that scorer quality directly limits policy learning.

3.6 Loss‑Curve Analysis

Huber loss declines smoothly from 0.18 to 0.04, while hinge loss shows a stair‑step decline with occasional rebounds, reflecting dense convergence signals from Huber and exploratory gradients from hinge.

3.7 Calibration vs. Ranking Pareto

Plotting Expected Calibration Error (ECE) against NDCG@5 yields:

MSE: ECE = 0.038, NDCG = 0.78

Hinge: ECE = 0.091, NDCG = 0.86

Mixed: ECE = 0.041, NDCG = 0.89

The mixed loss nearly dominates the Pareto frontier, achieving the best trade‑off between calibration and ranking.

3.8 Comparison with Standard Contrastive Losses

InfoNCE and Triplet losses, evaluated with the same scorer architecture, attain 77.8% and 77.1% RM‑Bench scores respectively—both below the mixed loss’s 79.2% because contrastive objectives focus on representation learning rather than absolute score calibration.

Reason Only When Needed: Efficient Generative Reward Modeling via Model-Internal Uncertainty
https://arxiv.org/abs/2604.10072
腾讯混元 & UNSW