How Quantization, Pruning, and Distillation Shrink AI Models for Edge Devices

Model Quantization

Core principle : Convert high‑precision parameters (e.g., FP32) to lower‑precision formats (e.g., FP16, INT8) while tolerating the resulting noise, thereby reducing storage and compute.

Key methods :

Post‑Training Quantization (PTQ) : Directly quantize a trained model without retraining; simple but may cause noticeable accuracy loss, especially below INT8.

Quantization‑Aware Training (QAT) : Simulate quantization errors during training so the model adapts; yields small accuracy drop (INT8 retains >95% performance) and suits high‑precision tasks.

Effect & scenarios : FP32→INT8 reduces model size by ~75% and speeds up inference 2‑4×; typical for mobile AI (real‑time beautification, speech recognition) and embedded devices.

Model Pruning

Core principle : Remove redundant parameters (e.g., near‑zero weights or entire channels) to shrink the network without significantly harming performance.

Key methods :

Unstructured pruning : Delete individual weights below a threshold, producing sparse matrices; high compression (50‑90%) but hard to accelerate on standard hardware.

Structured pruning : Remove whole structural units such as convolution kernels, channels, or attention heads; hardware‑friendly with moderate compression (30‑60%).

Effect & scenarios : Structured pruning cuts 40‑60% compute and 30‑50% model size (e.g., ResNet‑50 on edge devices); suitable for CNN‑based tasks like autonomous‑driving perception.

Knowledge Distillation

Core principle : A large teacher model guides a smaller student model to mimic its outputs and intermediate features, enabling the student to achieve performance close to the teacher despite a much smaller footprint.

Key methods :

Soft‑label distillation : Student learns the teacher’s probability distribution (soft labels), preserving inter‑class relationships.

Feature distillation : Student aligns its intermediate layer representations with the teacher’s.

Effect & scenarios : Student models can be 10‑100× smaller with < 3% performance loss (e.g., MobileBERT vs BERT‑base); ideal for NLP on phones or wearables.

Summary

Quantization excels at precision reduction and hardware friendliness, pruning focuses on removing redundancy and structural simplification, while distillation transfers knowledge to compact models; in practice they are often combined to achieve “tiny size, high performance, fast speed” on resource‑constrained devices.