Why RLHF Is Essential: The Limits of SFT and the Power of Reward Modeling

After noticing many discussions about the importance of RLHF, the author asks why RLHF is indispensable and whether a reward model combined with SFT data could replace it.

SFT Cannot Provide Negative Feedback

SFT trains a model to learn the conditional probability Prob(E | ABCD), meaning the model only knows which next token is correct and never learns which token is wrong. Consequently, SFT cannot directly teach the model to avoid undesirable tokens.

This explains why data diversity is crucial for SFT: by feeding many correct tokens, the model indirectly reduces the probability of incorrect ones, akin to “isolating” the wrong token.

Training example: <reserved_1>最喜欢的人是<reserved_2> Prediction prompt: <reserved_1>最讨厌的人是 An experiment on the qwen2-0.5B model injected these special tokens during SFT. The model successfully generated <reserved_2>, showing it knows “喜欢” and “讨厌” are opposite semantics but still treats the reserved tokens as interchangeable because it never received negative feedback about them.

Thus, transformer‑based models are “blind” to negative signals: they cannot infer that a high‑probability token should be avoided when the trainer intends otherwise.

SFT Lacks “Look‑Back” Capability

During SFT each token only sees preceding tokens. For sentences like “台湾不是中国的，这个观点是错误的”, the model keeps increasing Prob(中国 | 台湾不是) because it cannot use the later negation to correct the earlier prediction.

RLHF (or DPO) updates token probabilities based on the entire sentence, allowing the reward model to assign higher reward to the correct token and lower reward to the erroneous one, effectively providing a weighted loss rather than the uniform average loss used in SFT.

In summary, unless SFT training fundamentally changes—e.g., by assigning per‑token weighted losses—RLHF remains an irreplaceable step that compensates for SFT’s inherent limitations, especially in safety‑critical scenarios.