How a 1.5B Model Beats Cutting‑Edge Large Models on Math Exams

1. An Unexpected Phenomenon

When DeepSeek released the R1 series, it offered distilled versions that compress an 800 B teacher into models ranging from 1.5 B to 70 B. Benchmark results show the 32 B distilled model beating OpenAI’s o1‑mini, and the 1.5 B and 7 B versions scoring higher than GPT‑4o and Claude‑3.5‑Sonnet on AIME and MATH‑500 math contests.

2. Why Distillation Exists

Scaling model size usually improves capability, but larger models demand more memory, latency, and cost, making them unsuitable for edge devices or small teams. Knowledge distillation—first proposed by Bucila et al. (2006) and systematized by Hinton, Vinyals, and Dean (2015)—addresses the engineering problem of transferring a “high‑scoring student” model’s abilities to a lightweight “student” while tolerating some performance loss.

3. What Distillation Actually Transfers

Hard labels (one‑hot) convey only the correct answer, discarding relational information. A well‑trained teacher emits a probability distribution (soft label) that encodes similarity among classes—Hinton calls this “dark knowledge.” By raising the softmax temperature, the distribution is softened, exposing richer inter‑class structure that the student can learn.

Distillation passes not the answer itself but the probability structure hidden behind one‑hot labels.

The training objective combines two cross‑entropy losses: one matching the student to the teacher’s softened logits, the other matching the student to the true hard labels. Experiments show the softened loss usually needs higher weight, and when the student is much smaller, a lower temperature can be more effective.

Temperature acts as an information‑noise knob: raise it to release dark knowledge, but over‑raise injects noise.

4. Distillation Methods

Output distillation : aligns only final predictions (the most basic form).

Probability distillation : uses soft labels with temperature (Hinton et al., 2015).

Feature distillation : matches intermediate layer representations (FitNets, TinyBERT).

Attention distillation : aligns attention maps (Attention Transfer, MiniLM).

Relation distillation : preserves relative sample relationships (Relational KD).

Reasoning distillation : transfers chain‑of‑thought or step‑by‑step rationales (Distilling Step‑by‑Step, DeepSeek‑R1‑Distill).

Feature and attention distillation address the limitation of probability‑only methods by forcing the student to mimic the teacher’s internal computation, not just the final output.

5. Engineering Deployment

DeepSeek generated ~800 k high‑quality reasoning samples with the 671 B teacher, then fine‑tuned open‑source Qwen2.5 and Llama‑3 bases to produce six distilled models from 1.5 B to 70 B, all released under an MIT license and runnable on a single consumer‑grade GPU.

Official model cards report the 32 B version setting a new dense‑model record on multiple benchmarks, while the 7 B version achieved 55.5 % Pass@1 on AIME 2024, surpassing the open‑source QwQ‑32B‑Preview.

A contrast experiment showed that directly RL‑training a small model performed worse than distilling the teacher’s reasoning, highlighting the advantage of a strong teacher for inference capability.

Meta’s Llama 3.1 405 B also lists knowledge distillation as a core technique for its 8 B and 70 B variants, and OpenAI offers a “Stored Completions” product that automates the distillation pipeline for GPT‑4o mini.

6. Trade‑offs and Limits

Capability ceiling : Students cannot acquire abilities the teacher never demonstrated; on tasks like GPQA or LiveCodeBench the 1.5 B model lags behind GPT‑4o.

Student capacity : Excessive size gaps cause “capacity overload”; the Teacher‑Assistant approach (2019) inserts an intermediate model to bridge the gap.

Selective loss : Certain abilities (multi‑turn dialogue, tool use, long context) degrade after distillation.

Teacher quality : Errors, biases, or hallucinations in the teacher propagate to the student because distillation copies the entire probability distribution.

Distilled students can only approximate the teacher within the training data’s coverage; unseen abilities are not magically acquired.

7. Comparison to Human Knowledge Transfer

Both AI distillation and human teaching aim to move high‑cost expertise to a lower‑cost learner, including the reasoning process, not just the final answer. However, AI distillation uses a precise, differentiable loss (KL‑divergence or MSE) on shared vector spaces, whereas human mentorship relies on language, demonstration, and feedback, which are far less quantifiable.

Thus the high‑level idea of compressing knowledge is shared, but the mechanisms differ fundamentally in controllability, reproducibility, and scalability.

8. Conclusion

From Hinton’s 2015 temperature trick to DeepSeek’s 800 k‑sample pipeline that compresses a 671 B teacher into a 1.5 B student, distillation consistently trades a bounded performance drop for orders‑of‑magnitude cost reduction. Success hinges on three engineering balances: teacher capability ceiling, student capacity, and task distribution. Understanding these balances and tuning temperature, data quality, and distillation type yields far more reliable gains than blindly copying hyper‑parameters from papers.