Confidence‑Gated Reflection Boosts Reward Model Accuracy and Efficiency (CAMEL)

Background

Reward models (RMs) are central to aligning large language models (LLMs) in RLHF/RLAIF pipelines, providing human‑preference signals for candidate responses. Existing approaches fall into two camps: scalar RMs, which are fast and stable but offer limited interpretability, and generative judges, which produce detailed reasoning but incur high token and latency costs.

Recent work on confidence‑based computation allocation highlights the need to focus inference budget on truly uncertain samples. CAMEL (Confidence‑Gated Reflection for Reward Modeling) applies this principle to reward modeling by letting the model first assess its own certainty before deciding whether to invoke reflection.

Method: Confidence‑Gated Reflection

Step 1 – Single‑token initial verdict : Given a query and two candidate answers, the model outputs an initial verdict token ([[A]] or [[B]]). CAMEL does not train a separate confidence estimator; instead it uses the log‑probability margin between the two verdict tokens. A large margin indicates high confidence, while a small margin signals a difficult, ambiguous sample.

Step 2 – Confidence‑gated reflection : If the margin exceeds a preset threshold, the initial verdict is accepted directly (CAMEL‑Fast). If the margin is low, the model enters a reflection phase, re‑examining the candidates with criteria such as safety, relevance, and completeness before issuing a final verdict (CAMEL‑Reflection). This design yields the efficiency of scalar RMs for easy cases and the thoroughness of generative judges for hard cases.

Training Enhancements

To prevent the reflection step from merely echoing the initial judgment, CAMEL introduces Counterfactual Prefix Augmentation : for each training example, two versions are created that force the initial verdict to be A and B respectively. Training with GRPO (Generalized Reward‑Based Preference Optimization) rewards only the correctness of the final verdict, encouraging the model to overturn incorrect initial judgments during reflection without requiring extra human‑written explanations.

Experiments

CAMEL is built on Qwen3‑14B and trained on the Skywork Reward Preference 80K dataset. It is evaluated on three major reward‑model benchmarks: RewardBench , RM‑Bench , and JudgeBench , covering chat, safety, math, and code tasks.

Results:

CAMEL‑Fast achieves 90.5% (RewardBench), 74.8% (RM‑Bench), and 65.2% (JudgeBench).

CAMEL‑Reflection reaches 92.8%, 84.2%, and 71.6% respectively.

Overall average accuracy is 82.9% , a 3.2% improvement over the previous best baseline.

The 14B‑parameter CAMEL surpasses several 70B‑scale reward models such as LLaMA‑3.1‑Nemotron‑70B and INF‑ORM‑LLaMA3.1‑70B.

Analysis

Confidence scores derived from the log‑probability margin reliably separate easy from hard samples: high‑confidence regions contain most correct judgments, while errors concentrate in low‑confidence regions. By routing only low‑confidence samples to reflection, CAMEL reduces unnecessary token consumption, offering a clear accuracy‑cost Pareto frontier.

Training induces a “confidence shift”: the median confidence drops from 23.2 to 5.9, indicating the model becomes more conservative after learning to identify the truly decisive token for the final verdict.

Reflection provides measurable correction power: on RM‑Bench, reflection fixes 1,565 initial‑verdict errors while only overturning 332 correct ones, yielding a net gain of +1,233 corrected samples; RewardBench shows a similar positive net gain (+77).

Adjusting the confidence threshold allows seamless interpolation between CAMEL‑Fast and CAMEL‑Reflection, enabling deployment choices that balance throughput, latency, and accuracy. Compared with strong generative judges like RM‑R1‑DeepSeek‑32B, CAMEL‑Fast attains comparable performance using just one token.

Conclusion

CAMEL demonstrates that reward models need not sacrifice efficiency for expressiveness. By exploiting the log‑probability margin as a zero‑cost confidence signal, it delivers lightweight initial judgments and reserves costly reflection for genuinely difficult cases, achieving state‑of‑the‑art accuracy with a modest 14B model and offering a flexible accuracy‑cost trade‑off that aligns with emerging research on selective computation.