Why Self‑Distillation Is the 2026 Keyword for Continual Learning in Large Models

At the start of 2026, researchers in the large language model (LLM) field have converged on a common theme: self‑distillation. The most discussed arXiv papers all revolve around this technique, which promises a path to continual learning without external strong teachers.

1. Self‑Distillation Enables Continual Learning

The paper “Self‑Distillation Enables Continual Learning” (https://www.alphaxiv.org/abs/2601.19897, code: https://github.com/idanshen/Self-Distillation) identifies catastrophic forgetting as a major drawback of supervised fine‑tuning (SFT). It proposes a Self‑Distillation Fine‑Tuning (SDFT) method that first constructs a few‑shot context to elicit a high‑quality teacher distribution from the model itself, then trains the model to match this distribution without demonstrations. By treating continual learning as an on‑policy alignment problem, the approach preserves the original probability flow and avoids the severe parameter drift that causes forgetting. Experiments on skill‑learning and knowledge‑acquisition tasks show higher accuracy on new tasks and a marked reduction in forgetting, allowing a single model to accumulate multiple skills over time.

2. Reinforcement Learning via Self‑Distillation

The second paper, “Reinforcement Learning via Self‑Distillation” (https://arxiv.org/pdf/2601.20802, code: https://github.com/lasgroup/SDPO), observes that standard reinforcement learning methods such as GRPO receive only binary feedback, which leads to credit‑assignment problems and stagnation when rewards are zero. The authors introduce the Self‑Distillation Policy Optimization (SDPO) framework, which converts sparse scalar rewards into rich token‑level supervision. By feeding error messages back into the context as a self‑reflective teacher, SDPO generates dense learning signals that pinpoint failing tokens. In hard tasks, SDPO reaches comparable solution discovery rates with roughly one‑third the number of attempts (3× speed‑up) and solves about 70 % of difficult problems at k = 1000, outperforming traditional algorithms. On the LiveCodeBench benchmark, SDPO attains the same accuracy with only one‑quarter of the generated samples required by GRPO.

3. Self‑Distilled Reasoner

The third work, “Self‑Distilled Reasoner: On‑Policy Self‑Distillation for Large Language Models” (https://arxiv.org/pdf/2601.18734), tackles the challenges of large search spaces and sparse rewards in complex reasoning. It proposes the On‑Policy Self‑Distillation (OPSD) framework, which creates two policy states: a teacher policy that receives privileged information (e.g., ground‑truth answers) and a student policy that answers without such information. Training minimizes the KL divergence between student and teacher token distributions, forcing the model to align its internal distribution with the high‑quality teacher. Experiments on MATH and GSM8K show a 4–8× improvement in token utilization over GRPO, and OPSD extracts additional reasoning potential beyond what SFT provides.

All three papers share a common logic: they exploit the model’s existing in‑context learning ability to generate an information asymmetry that drives self‑supervised improvement. Self‑distillation is thus emerging as a standard post‑training technique for enabling continual learning in large models.