How a Simple Colon Can Trick Top LLMs – The Master‑RM Fix

Universal “Key” That Can Fool LLM Reward Models

Recent work shows that using large language models (LLMs) as evaluators in reinforcement learning with verifiable rewards (RLVR) is increasingly common, but a tiny token such as a colon can make these evaluators issue false‑positive rewards, effectively opening a backdoor.

Both non‑textual symbols (e.g., spaces, ".", ",", ":") and reasoning prefixes (e.g., "Thought process:", "Solution", "解") are sufficient to trigger the bug. The issue affects major LLMs such as GPT‑4o, Claude‑4, and LLaMA3‑70B, which all fail when presented with these cues.

The "universal key" can be divided into two categories:

Non‑textual symbols : spaces, ".", ",", ":".

Reasoning prefixes : strings that only indicate the start of a reasoning process, such as "Thought process:", "Solution", "Let’s solve this problem step by step", without containing substantive content.

To assess the prevalence of this reward‑model deception, researchers evaluated a range of LLMs on multiple datasets and prompt formats. Two model groups were tested: specialized generative reward models (e.g., Multi‑sub RM, Omni‑Judge) and general‑purpose LLMs (e.g., GPT‑4o, Claude‑4, LLaMA3‑70B, Qwen2.5‑72B). Ten adversarial responses were crafted, including the symbols and multilingual reasoning prefixes, and five reasoning benchmarks (both general and mathematical) were used.

Experimental results show that every tested model is vulnerable: GPT‑4o yields a 35% false‑positive rate (FPR) for the symbol ":", LLaMA3‑70B reaches 60‑90% FPR for "Thought process:", and the proprietary General‑Verifier attains 66.8% FPR for spaces on the MATH dataset. The phenomenon is language‑agnostic, appearing equally in Chinese and Japanese prompts.

Further analysis of the Qwen2.5 series (0.5B–72B) indicates that model size does not monotonically reduce vulnerability. Smaller models rely on literal matching with low FPR but poor consistency; mid‑size models (1.5B–3B) detect semantic similarity but suffer higher FPR; 7B–14B models achieve the lowest FPR with good consistency; the largest models (32B–72B) again show increased FPR because they prefer solving problems themselves rather than comparing responses.

A “Judge” Model That Won’t Be Fooled

To mitigate the universal‑key effect, the authors built a new reward model called Master‑RM (Master Reward Model). They sampled 20 k adversarial examples from the original 160 k training set, generated responses with GPT‑4o‑mini that contain reasoning prefixes, kept only the first meaningless sentence, and labeled them as “incorrect”. These adversarial samples were merged with the original data to form an enhanced training set.

Master‑RM was then fine‑tuned on Qwen2.5‑7B‑Instruct using supervised fine‑tuning (SFT) to minimize cross‑entropy loss, teaching the model to distinguish genuine answers from surface‑level deceptive ones.

When re‑evaluated under the same conditions, Master‑RM achieved near‑zero false‑positive rates (0 % across all universal‑key attacks) and demonstrated strong robustness on unseen datasets and attacks. Its evaluation consistency with GPT‑4o reached 0.96, confirming its effectiveness as a general‑domain generative reward model.

The study highlights the fragility of LLM‑based judges—an innocuous colon can cause them to err—and stresses the need for rigorous adversarial evaluation in RLHF pipelines.