Why Step-Level DPO Is Revolutionizing LLM Math Reasoning

Recent small‑scale LLMs (7B) have achieved large gains on GSM8K and MATH by using step‑level DPO, a variant of preference‑based optimization that constructs a partial‑order dataset from step‑chosen and step‑rejected pairs. Unlike instance‑level DPO, which uses full trajectories, step‑level DPO optimizes only the step‑level data while treating the common prefix as part of the prompt and excluding it from loss computation.

Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learning

The paper introduces step‑level DPO and obtains step‑level partial‑order data by running tree search to generate trajectories that share a common prefix. Tree search naturally yields such prefixes, and the authors apply UCT, estimated‑Q, and other MCTS metrics to select preference steps. They also perform label smoothing on the DPO loss based on visit counts.

Step-level Value Preference Optimization for Mathematical Reasoning

This work builds on AlphaMath by combining value‑function estimation with step‑level DPO. Preference data are created similarly to the first paper—using tree search plus an output‑reward filter to select chosen and rejected steps. During training a value head is added, enabling value‑guided decoding whose sampling cost lies between greedy/random sampling and full MCTS, yielding better results. An additional SFT loss is introduced to prevent model degradation.

RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight‑Fold

The authors systematically study how using incorrectly answered synthetic data can improve mathematical reasoning. They employ a simple rollout to estimate a value function for each step, then select chosen and rejected steps based on Q‑values. When optimized with step‑level DPO, the same accuracy is achieved with eight times less data.

Step‑DPO: Step‑wise Preference Optimization for Long‑chain Reasoning of LLMs

The authors consider this the most solid recent work. Experiments on GSM8K and MATH across many base, instruction‑tuned, and RL‑trained models show consistent improvements, even for larger models such as DeepSeekMath‑RL, Qwen‑2 variants, and LLaMA‑3 (7B‑70B). Preference datasets are built by first prompting DeepSeek‑Math‑Instruct on Metamath, MMIQC, etc., filtering for correctly answered responses, and then constructing step‑wise pairs using a prompt format different from standard SFT datasets.

Our Practice

We previously evaluated TDPO on the Eurus platform. Here we reproduce experiments on the Step‑DPO‑10k preference dataset and compare TDPO with step‑level and instance‑level DPO when the common prefix is placed either in the prompt or the response. TDPO uses the same hyper‑parameters as the original papers (dpo‑beta=0.4). The dataset provides both (step‑chosen, step‑rejected) and (full‑chosen, full‑rejected) pairs, allowing us to train four configurations: step‑level DPO, instance‑level DPO, step‑level TDPO, and instance‑level TDPO.

Results show that our re‑implemented trainer matches the official inference outputs. Changing the template (e.g., using an Alpaca‑style prompt) causes a modest drop for both step‑DPO and TDPO, with TDPO degrading more noticeably. Nevertheless, with the Alpaca template TDPO reaches an 87.11% solve rate on GSM8K, while step‑DPO yields larger gains on MATH.

Both the Eurus‑Preference‑Dataset and the Step‑DPO‑Preference‑Dataset improve SFT models to varying degrees. Algorithms such as step‑DPO and TDPO provide stable enhancements without causing capability regression. In contrast, XPO experiments based solely on SFT‑generated preference pairs tend to diverge quickly, especially for math and code tasks.

When constructing new preference datasets from different prompts or model responses (e.g., Eurus‑Preference‑Dataset, Step‑DPO‑Preference‑Dataset), XPO training becomes more stable. These out‑of‑distribution datasets appear to benefit offline‑RL methods, whereas in‑distribution data (derived from the same SFT model) can restrict optimization and increase the risk of breaking the original distribution, particularly for tasks with strict format constraints like math and code.

References

Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learning

Step‑level Value Preference Optimization for Mathematical Reasoning

RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight‑Fold

Your Language Model is Secretly a Q‑Function