Can Intuitive Fine‑Tuning Replace Expensive RLHF and DPO for LLM Alignment?

Background

Large language models (LLMs) like ChatGPT have become ubiquitous, but they still suffer from hallucinations and poor adherence to complex instructions, limiting their practical value.

Typical LLM Training Pipeline

The prevailing three‑step pipeline consists of:

Pre‑Training (PT) : learns grammar, logic and world knowledge from massive corpora but does not guarantee alignment with human values.

Supervised Fine‑Tuning (SFT) : continues PT using higher‑quality, instruction‑style data, yet still predicts the next token based only on ground‑truth context.

Preference Optimization (PO) : aligns the model with human preferences, the most common implementation being Reinforcement Learning from Human Feedback (RLHF).

Why SFT, RLHF and DPO Differ Only in Preference Estimation

All three methods first estimate the model’s own preferences and then align them with human preferences. SFT estimates preference from a single predicted token, while RLHF and Direct Preference Optimization (DPO) estimate from an entire generated sentence, yielding more accurate preference signals but at a much higher cost.

RLHF: Benefits and Drawbacks

RLHF trains a reward model (RM) on human‑ranked responses and then uses Proximal Policy Optimization (PPO) to fine‑tune the LLM. This approach dramatically improves alignment but requires:

Data construction : multiple responses per instruction and human ranking, which is labor‑intensive.

Compute resources : on‑the‑fly generation and scoring of responses during training, often needing 3‑4 LLM copies on a GPU simultaneously.

Moreover, the mismatch between PT/SFT objectives and the true generation objective can cause instability and catastrophic forgetting.

Direct Preference Optimization (DPO)

DPO merges the reward model and policy model into a single training step, reducing the need for a separate reference model. However, the ideal “online DPO” still requires real‑time generation and human ranking, so data‑construction costs remain high.

Offline DPO

Offline DPO collects model responses in advance, ranks them offline, and then fine‑tunes. While cheaper than online DPO, it still suffers from the same drift problem as RLHF: as training progresses the model’s preferences diverge from the collected data, leading to biased loss estimates.

Intuitive Fine‑Tuning (IFT)

IFT proposes a unified view: instead of treating SFT, RLHF and DPO as separate stages, it introduces a Temporal Residual Connection (TRC) that feeds the model’s own previous prediction back into the context for the next prediction. This soft‑sampling strategy approximates the full‑sentence preference estimation of RLHF/DPO while keeping the computational cost of SFT.

Key benefits of IFT:

Better alignment of the training objective with the real generation process because the model sees its own predictions during training.

Enhanced causal reasoning and factuality via Dynamic Relation Propagation (DRP) , which models how a current token influences all future tokens.

The underlying objective can be expressed as an expectation over the model’s own token distribution:

E_{\rho\sim D}[\log P_{\theta}(w_{t+1}\mid w_{\le t})]

where the distribution \(\rho\) is the model’s own prediction rather than the ground‑truth token.

Other Related Techniques

Scheduled Sampling

Gradually replaces ground‑truth tokens with model‑generated tokens during training, moving the model toward the real generation distribution. IFT can be seen as a soft‑sampling evolution of this idea.

Noisy Embedding Fine‑Tuning

Injects random noise into input embeddings to improve robustness. IFT’s TRC adds a *causal* noise that preserves context, offering stronger factual consistency than random perturbations.

References

For detailed derivations and experimental results, see the original papers:

Ziegler et al., “Fine‑tuning language models from human preferences,” arXiv:1909.08593, 2019.

Ouyang et al., “Training language models to follow instructions with human feedback,” NeurIPS 2022.

Schulman et al., “Proximal policy optimization algorithms,” arXiv:1707.06347, 2017.

Rafailov et al., “Direct Preference Optimization: Your language model is secretly a reward model,” NeurIPS 2024.

Hua et al., “Intuitive fine‑tuning: Towards simplifying alignment into a single process,” arXiv:2405.11870, 2024.

Bengio et al., “Scheduled sampling for sequence prediction with recurrent neural networks,” NeurIPS 2015.

Jain et al., “Neftune: Noisy embeddings improve instruction finetuning,” arXiv:2310.05914, 2023.