Model Distillation: Teaching Small Models to Learn from Large Ones

Introduction

A technical Q&A application often needs to rewrite user questions before retrieval and answering. While a large model can handle this, using it for every step is costly and slow. Model distillation offers a way to teach a smaller model a specific task, such as question rewriting.

Distillation borrows from chemistry: separating components by volatility. In AI, it means extracting the capabilities a student model needs from a teacher model's demonstrated behavior. Knowledge distillation typically uses teacher outputs, probability distributions, or other signals to train the student. The goal is to retain task-specific ability at lower deployment cost, not to compress the teacher's weights or copy all its capabilities. Success depends on task scope, training data, and method.

Teaching a Small Model a Specific Task

1. Define What the Student Should Learn

"Make the small model as smart as the large one" is too vague. For question rewriting, concrete goals include:

Preserve the user's original intent; do not add information the user did not provide.

Reorganize wording, remove irrelevant conversational filler.

Only rewrite the question; do not answer it.

When information is missing, keep the uncertainty; do not invent device or software versions.

Clear goals prevent drift during data preparation and evaluation.

2. Let the Teacher Provide Demonstrations

Collect a batch of real user questions (with privacy removed) and supplement with handcrafted cases for uncovered expressions. The teacher prompt can be:

Rewrite the user's conversational question into a clear technical question.
Preserve the original intent; do not add facts the user did not provide; do not answer the question.
If the original text does not explicitly mention device, version, or cause, do not fill them in.
Output only the rewritten question.

For example, user input:

The model on my computer takes forever to start outputting each time I ask it,
but once it starts the speed is normal. What could be the reason?

The teacher's demonstration output:

The local model has a long wait time for the first token after each request,
but generation speed is normal once started. What are possible causes?

This yields input–target pairs. Training data usually includes the task instruction; the exact format (chat, completion) depends on the training tool. Note that we collect teacher demonstrations via API, not by extracting weights. Service terms and data licenses still apply.

3. Check Demonstrations Before Training

Teachers make mistakes. For instance, the teacher might rewrite the above question as "Why is my VRAM insufficient?" even though the user never mentioned VRAM. Such demonstrations look professional but alter the task. Generated data must be filtered: check for added facts, direct answers, or lost intent; remove near-duplicates; ensure the set includes already-clear questions and genuinely ambiguous ones. Otherwise the student may learn to over-rewrite everything. Hold out a test set that does not overlap with training data to avoid measuring memorization.

4. Train the Student with These Demonstrations

Choose a base student model with suitable foundational ability and language support. Training can save full model weights or only adapter weights (e.g., LoRA). This example continues training an existing model, not building from scratch.

Supervised fine-tuning (SFT) is used: the student generates an output for each input, the training program compares it to the target, and updates trainable parameters. After many iterations, the student should apply the same rules to unseen questions.

This differs from pasting examples into a chat window: in-context examples guide the model only for the current session; training updates parameters so the behavior persists.

A natural question: since both involve continued training, what distinguishes distillation from fine-tuning?

Fine-tuning describes the training process: continuing training on an existing model to adapt to new tasks, knowledge, or output formats. SFT uses labeled input–target pairs.

Distillation focuses on the learning relationship and signal source: the teacher provides answers, probability distributions, or intermediate representations, and the student learns from them. SFT is a common training method for distillation.

Overlap exists:

Training with human-written Q&A pairs is SFT, not usually called distillation.

Using teacher-generated answers as targets for SFT is both fine-tuning and distillation.

If the student also learns the teacher's probability distributions or intermediate representations, it is distillation with a different training objective.

In short: Fine-tuning describes how the model continues training; distillation describes what the student learns from the teacher. Not all fine-tuning is distillation, and distillation requires a concrete training method.

5. Test with New Questions Before Deployment

After training, evaluate the student on unseen questions, comparing against the teacher and the pre-training student. For question rewriting, check: intent preservation, no added facts, adherence to "rewrite only" rule. Then compare latency and deployment cost. A smaller model may be cheaper but regress on complex phrasing. Saving API costs while corrupting user queries is not a win. Remedies: add more data and retrain, or route uncertain requests to a stronger model. Once deployed, the student typically runs independently without calling the teacher.

Why Teacher Demonstrations Can Teach the Student

The training process does not embed the teacher's identity into the student; it adjusts the student's parameters so that on similar inputs it tends to produce outputs matching the demonstrations. If demonstrations are well-chosen and training effective, the student may learn patterns: which conditions must be kept, how to restructure sentences, why not to invent missing information. Generalization to new questions must be verified by testing. Text demonstrations are one distillation approach. Others have the student learn the teacher's probability judgments over alternatives (soft labels). For example, in image classification, telling the student "this is a cat" versus "most likely cat, somewhat like a fox, almost certainly not a car" conveys inter-class similarity. Classic knowledge distillation research uses such richer signals. LLM distillation can leverage generated answers, reasoning steps, output probabilities, or intermediate representations. Teacher-generated explanations are learnable demonstrations, but they do not necessarily reveal the teacher's true internal reasoning. The student's learning is bounded by its own capacity, prior ability, and training data coverage. Excelling on one task does not imply equal performance on others.

Comparing Distillation, Fine-Tuning, and Quantization

The previous article covered quantization. Viewing all three together clarifies their distinct questions:

Distillation : Where do learning signals come from? Uses teacher-provided signals to teach the student needed capabilities.

Fine-Tuning : How does a model adapt to a new task? Continues training on an existing model, updating parameters or adapters.

Quantization : How are model numbers represented? Represents weights and other values with lower precision or more compact formats.

They can be combined: a single training run can be both distillation and fine-tuning; after training, the student can be further quantized to reduce storage and runtime memory. Quantization usually does not reduce parameter count; distillation can use a smaller student but does not simply delete a subset of the teacher's parameters.

Why Distillation Results May Still Be Unsatisfactory

Demonstrations Look Correct but Change Task Requirements

Teachers err. If the teacher frequently adds unstated causes in question rewriting, the student may imitate that. Generating 10,000 samples does not guarantee 10,000 valid demonstrations. Filtering must follow task standards, not just fluency. Erroneous demonstrations and excessive repetition degrade training.

Training Questions Work, but Paraphrases Fail

If data only contains simple, complete questions, the student will struggle with colloquial expressions, missing information, or ambiguity. Memorizing a few patterns is insufficient for real users. Data must cover varying difficulty and expression styles, including cases that need no rewriting and cases where meaning cannot be determined. Testing on held-out questions reveals whether the student learned a method or just memorized patterns.

Student Capacity Limits or Overly Broad Task Scope

Distillation cannot guarantee the student matches the teacher in all respects. The student's base capacity, size, and training method affect the outcome. Teaching it to only rewrite questions is a different difficulty level than requiring complex reasoning, coding, and expert Q&A simultaneously. If the task scope is too broad or hard requests often exceed the student's ability, reconsider model choice and task division. Evaluating whether distillation is worthwhile requires weighing training investment, inference cost, and latency—not just model size reduction.

Conclusion

Model distillation lets a student learn a capability from teacher-provided signals. Data preparation, demonstration filtering, student training, and testing on new questions are all essential. It is not packaging a large model into a smaller file. In practice, start with a well-bounded task and verify the student can handle it reliably. For question rewriting, preserving user intent and following output constraints matter more than making sentences sound "more professional."

Extension: From Model Distillation to Experience Distillation

In writing and agent discussions, "distillation" is also used metaphorically for experience extraction: organizing one's own or others' practices for reuse by AI or humans. This is a capability-transfer metaphor, not a full replication of knowledge and judgment. Style distillation is one example: extracting vocabulary habits, structural patterns, and content selection from articles and edit histories. To make an AI continue a personal blog style, unify team documents, or borrow a certain article's explanatory approach, collect representative samples (with rights), have the AI analyze language, structure, and selection habits; confirm stable rules with human review, add positive and negative examples, save as a style guide, prompt, or skill. Then test on a new topic and iterate. The goal is to learn the expression method, not copy the original text. Code review, debugging, and other experiences can similarly be structured into processes and checklists. However, this typically provides requirements to an existing model via context without updating model parameters, which is different from the model distillation training described above.