Understanding the Post-Training Process in DeepSeek V4‑Flash

Post‑Training in the Full Pipeline

Pre‑training predicts the next token after processing 320 trillion tokens, giving the model massive knowledge but no task orientation. Post‑training converts this "knowledge engine" into a "task executor" through three layers: supervised fine‑tuning (SFT), reinforcement learning (RL), and distillation.

SFT : trains on high‑quality Q&A and reasoning data using maximum‑likelihood to teach the model the desired format and basic behavior.

RL : lets the model generate answers and receives reward signals that indicate what counts as good, teaching the notion of quality.

Distillation : compresses the output distributions of one or more stronger models into the target model, effectively transferring capability.

GRPO: DeepSeek’s Core RL Algorithm

GRPO (Group Relative Policy Optimization) first appeared in the 2024 DeepSeekMath paper and remains the main RL method. Unlike standard PPO, which adds a separate value network, GRPO removes the value network and instead uses group‑wise reward baselines: for each question, a set of sampled answers is compared, and the advantage is computed by normalizing within the group.

The loss retains PPO’s clipped importance sampling and directly adds a KL‑divergence constraint to the loss function, keeping the model close to a reference model and preventing language‑ability collapse.

Note: later analysis (Liu et al., arXiv:2503.20783) questions the use of standard‑deviation normalization, suggesting it can bias simple versus hard tasks and proposing a variant without that term.

Reward Sources

In the R1 “reasoning‑focused RL” stage, rewards are rule‑based: correct answers for math, passing test cases for code, and penalties for format violations, deliberately avoiding neural reward models to prevent reward hacking. Later V4 introduces a generative reward model (GRM) that first produces an evaluation according to a rubric and then assigns a score, embedding reasoning into the reward.

The post‑training compute budget exceeds 10 % of pre‑training cost, and over 1 800 environments and 85 000 complex prompts were synthesized for RL.

V4 Shift: Expert‑Specialized Training + Online Policy Distillation

V4 splits post‑training into two stages. First, domain‑specific expert models (math, code, agents, instruction following) are trained via SFT followed by GRPO‑guided RL with custom reward models, sometimes further divided by reasoning budget (no‑think, high‑think, max‑think). This isolates capabilities and avoids “capacity fighting.”

Second, these experts are merged into a single student model using Online Policy Distillation (OPD). The student generates rollout trajectories; teachers score only the paths actually taken, and the loss is the reverse KL divergence between student and teacher distributions.

Two design choices are highlighted:

Reverse KL (seeking‑mode) penalizes only where the student assigns probability, allowing the student to ignore low‑probability teacher suggestions, which is crucial when many teachers disagree.

Full‑vocabulary logit distillation: instead of estimating KL on a single sampled token, V4 compares the entire vocabulary distribution at each position. To manage memory, teacher hidden states are cached, and a custom TileLang operator computes exact KL.

Similar “multiple‑expert RL + multi‑teacher online distillation” pipelines appear in Kimi K3, GLM‑5, and Nemotron‑Cascade 2, indicating a broader industry trend.

Returning to V4‑Flash

The July 31 V4‑Flash release shows a step‑wise score jump because the same architecture benefits from stronger experts, richer agent‑training data, and a more mature execution framework. Although no dedicated technical report for this specific version exists, the observed improvements align with the methods described in the V4 technical report.

Thus, while earlier consensus held that parameter count dominates performance, the current view (2026) is that, given a fixed architecture, post‑training now determines the practical ability frontier, driven jointly by algorithmic advances and engineering effort.