Why RL Matters: From Reinforcement Learning to (Soft) Distillation

Understanding cot patterns

In the supervised fine‑tuning (SFT) stage, the model learns chain‑of‑thought (cot) patterns that strongly influence the upper bound of performance in the subsequent post‑training stage. A cot pattern is a reasoning template that can be expressed in different ways. For the prompt 997 × 1003, three cot patterns are illustrated:

good pattern :

997 × 1003 = (1000 − 3)(1000 + 3) = 1000² − 3² = 999991

mediocre pattern :

997 × 1003 = 1003 × (900 + 90 + 7) = 902700 + 90270 + 7021 = 999991

bad pattern : 997 × 1003 = 999991 All three yield the correct answer, but their generalization differs: the good pattern is concise and reliable; the mediocre pattern involves many arithmetic steps, increasing error probability and reducing token efficiency; the bad pattern provides no reasoning trace and may mislead smaller models.

Why RL matters in post‑training

Good cot patterns produce shorter outputs, higher accuracy, and better generalization. While SFT supplies an initial cot pattern, reinforcement learning (RL) explores many responses to discover expressions that best match the model’s knowledge and tokenizer, effectively localizing the pattern for the target model.

In reasoning tasks, RL lets the model decide how many reflection steps are optimal, and in agent‑based tasks it determines which sub‑agents to invoke and when. Larger models (e.g., 1T parameters) can explore richer patterns than smaller ones (e.g., 100B), and RL helps each model find the most suitable agent configuration.

RL also improves controllability. Pre‑training injects noisy, heterogeneous patterns that SFT alone cannot suppress. After RL, models stop emitting excessive <think> / </think> tags, reduce language mixing, and produce more stable output lengths, because the reward signal penalizes undesirable behaviors.

RL versus distillation

There is an ongoing debate about whether RL or distillation matters more. The article notes that no direct evidence shows RL yields qualitatively superior models compared to SFT alone; the gains from RL can often be captured by distillation. By treating the RL‑trained model as a teacher, its explored high‑quality patterns can be transferred to a student model through distillation, achieving comparable metrics with far less compute.

Scaling observations: a 1T‑parameter model discovers richer patterns than a 100B‑parameter model; distilling those patterns into the smaller model narrows the performance gap. This approach works especially well when the student and teacher share the same vocabulary.

Soft distillation

Soft distillation goes beyond copying raw responses; it leverages a strong teacher to generate data that highlights pattern deficiencies, synthesizes boundary‑case examples, and refines the verifier used during RL. By selectively keeping high‑quality English cot patterns and discarding lower‑quality ones, soft distillation can produce a student that outperforms its teacher.

The article concludes that believing distillation is useless reflects a lack of hands‑on large‑model training, while treating distillation as the sole post‑training strategy suits teams that prioritize efficiency over frontier performance.