Can You Claim to Know Large Models? Guide to Distillation, Quantization & Fine‑Tuning

DeepSeek V3/R1 has attracted worldwide attention, but its 671 B parameters (requiring 450‑1400 GB of VRAM) make local deployment challenging. To make large models more practical, researchers apply three techniques: distillation, quantization, and fine‑tuning.

1. Distillation: Teacher’s Knowledge, Student’s Speed

Model distillation, proposed by Hinton et al. (2015), transfers knowledge from a large “teacher” model to a smaller “student” model by reducing model size or migrating knowledge.

(a) Structural Decomposition

Large models consist of many components such as attention layers and fully‑connected layers. By removing less‑important structures (e.g., structure C) while keeping critical ones (A, B, D), the student model becomes smaller without severe performance loss. This process is related to the research area of model pruning.

(b) Knowledge Transfer

Another distillation approach uses the teacher’s soft‑label probability distribution to train the student. The workflow is:

Choose a powerful teacher model (e.g., DeepSeek‑R1 or DeepSeek‑V3).

Generate soft labels by feeding diverse question sets (math, finance, computer science, etc.) into the teacher.

Train the student model on this generated dataset, allowing it to inherit the teacher’s capabilities.

2. Quantization: From Float to Integer, Lightening the Load

Quantization reduces the bit‑width of floating‑point matrices that dominate a model’s memory and compute. For example, converting FP16/FP8 weights to INT4 cuts storage size dramatically.

Illustration of floating‑point vs. reduced‑precision storage:

Quantization speeds inference and lowers deployment cost, but the reduced precision can increase error on complex tasks, potentially degrading performance.

3. Fine‑Tuning: Small Adjustments for Task‑Specific Excellence

Fine‑tuning adapts a generally capable large model to specialized domains (legal, financial, etc.) by further training on domain‑specific data.

Three fine‑tuning strategies are described:

Full‑parameter fine‑tuning : updates all model weights; yields strong results but requires abundant data and compute, and may overfit on small datasets.

Partial fine‑tuning : freezes most parameters (e.g., structures A, B, C) and updates only a subset (e.g., structure D), suitable for limited resources.

Adapter‑based fine‑tuning : inserts small adapter modules (e.g., LoRA) and trains only these, leaving the original weights untouched; effective when data or compute are scarce.

Conclusion: Three Masters, Each with Strengths

Distillation excels at knowledge transfer, quantization at size reduction, and fine‑tuning at task‑specific performance. Depending on the scenario—lightweight deployment or specialized accuracy—practitioners can choose one technique or combine them to build a compact yet powerful AI assistant.

If you need a lightweight model, consider distillation or quantization.

If you need superior performance on a specific task, fine‑tuning is the preferred approach.