MIPU: New RL Method Bridges LLM Training-Inference Gap

Background: Training-Inference Mismatch in LLM RL

In large model reinforcement learning training, optimization during training does not always translate into stable capability improvements at inference time. Researchers typically adjust reward functions, policy updates, sampling methods, and optimization objectives, judging progress by training loss, reward changes, or policy convergence. However, training involves frameworks like vLLM, SGLang, Megatron, and FSDP, whose differences in sampling, precision, parallelism, and implementation details can cause divergent behavior between training and inference. This phenomenon is known as training-inference mismatch .

The core question becomes: should we continue optimizing the training process itself, or directly optimize how the model generates final answers?

MIPI Principle: Redefining Training Optimization Goals

Traditional methods focus on improving training strategies (learning rate, sampling, reward design, policy updates). MIPI shifts the focus: it directly incorporates the quality of the model's actual generated answers into the training optimization objective. The principle states that a training update has real value only if, after each update, the model produces higher-quality, more stable answers in real inference tasks.

Let μₖ denote the inference policy after the k-th update, πₖ the training policy used at the k-th round, and J the evaluation function on actual inference tasks. MIPI requires: J(μₖ₊₁) − J(μₖ) ≥ 0 That is, inference performance should not decrease after each update. The change in inference performance can be decomposed as:

J(μₖ₊₁) − J(μₖ) = [J(μₖ₊₁) − J(πₖ₊₁)] + [J(πₖ₊₁) − J(πₖ)] + [J(πₖ) − J(μₖ)]

The first term is the gap between updated inference and training policies, the second is the change from training policy update, and the third is the pre-update gap. This decomposition shows that final inference performance depends not only on training policy optimization but also on the alignment between training and inference policies.

MIPU: Two-Step Implementation

To operationalize MIPI, the authors propose MIPU, which organizes training into two main steps:

Step 1: Generate and Filter Training Candidates

In methods like GRPO, the model generates answers under the new training policy and updates parameters based on reward signals. MIPU simultaneously considers the probability changes of candidate answers under both the current and updated training policies. By comparing policy changes before and after the update, MIPU gains more information about whether the update truly alters inference behavior. The implementation handles importance weights and clipping to reduce instability from large weight changes.

The importance weight ratio is computed as: πᵏθ / μₖ = (πₖ / μₖ) × (πθ / πₖ) After Step 1, MIPU obtains a set of candidate models or updates for further evaluation.

Step 2: Evaluate Candidates and Decide Updates

MIPU evaluates the candidate models' generated answers. If the updated model assigns higher probability to high-quality answers and lower probability to low-quality ones, the update is considered valuable. Conversely, if the update only changes the answer distribution without improving quality, it may be rejected. The goal is not to reject as many updates as possible, but to balance performance improvement with necessary exploration; rejecting too many candidates can stall the model in old parameter states.

Experimental Setup

Experiments were conducted on Qwen3-1.7B and Qwen3-4B models using FP8 training , evaluating on math reasoning benchmarks: MATH-500, AIME24, OlympiadBench, Minerva, AMC23 . MIPU was compared against standard GRPO, MIS, and learning rate decay . Metrics included pass@1, avg@16, pass@16, average accuracy across model sizes, and training stability.

Results

MIPU achieved higher average accuracy than standard GRPO on both model sizes:

Qwen3-4B : 66.71% average accuracy

Qwen3-1.7B : 53.97% average accuracy

Beyond final scores, MIPU exhibited more stable score trajectories during training.

Ablation analysis showed that candidate generation alone does not reliably select effective updates; the candidate evaluation step (Step 2) is crucial for identifying valuable updates. The full MIPU (generation + evaluation) achieved the highest average accuracy. The results confirm that candidate filtering is not about rejecting more updates, but balancing acceptance of effective updates with retention of necessary exploration.

Conclusion and Future Work

This work addresses the training-inference inconsistency in LLM RL training by proposing the MIPI principle and its concrete implementation MIPU. The core insight: training optimization should not only improve the training process but must verify that improvements actually enhance final inference performance. MIPU achieves this by generating candidates, evaluating them on real tasks, and filtering updates based on inference quality.

In FP8 experiments, MIPU delivered higher average accuracy and more stable training trajectories on Qwen3-1.7B and Qwen3-4B. Current experiments are limited to 1.7B/4B scales and math reasoning tasks. Future work includes testing MIPU on larger models, code generation, knowledge QA, tool use, and across different training frameworks and inference systems.