Compression Techniques for BERT: Analysis, Quantization, Pruning, Distillation, and Structure-Preserving Methods

The rapid growth of BERT‑style Transformer models has led to ever‑larger parameter counts, making deployment on high‑end GPUs necessary; however, mobile and edge devices demand compressed models due to limited compute and storage.

BERT Model Analysis : BERT consists of an embedding layer, a linear‑before‑attention layer, multi‑head attention, a linear‑after‑attention layer, and a feed‑forward layer. The three hyper‑parameters L (number of Transformer blocks), H (hidden size), and A (number of attention heads) determine model width, depth, and attention diversity. Empirical measurements on NVIDIA Titan X show that the feed‑forward layer consumes roughly half of the storage and computation, while the embedding layer occupies a large portion of storage but little compute, and the multi‑head attention layer uses minimal storage yet non‑trivial compute.

Quantization : Reducing weight precision (e.g., from fp32 to fp16) halves storage and can double inference speed on compatible hardware. Quantization is generally friendly to fully‑connected layers, but the embedding layer is sensitive and should often be excluded. Quantization‑aware training (QAT) mitigates accuracy loss.

Pruning : Two main types are element‑wise pruning (sparsifying individual weights) and structured pruning (removing entire heads or layers). Element‑wise pruning works well for fully‑connected layers, while structured pruning can target attention heads (e.g., reducing 12 heads to 4) or entire Transformer blocks based on importance metrics such as loss impact or L1 norm. Pruning can be performed during training or post‑training, with tool support in TensorFlow Model Optimization Toolkit and PyTorch torch.nn.utils.prune .

Knowledge Distillation : A teacher‑student framework transfers knowledge from a large teacher model (Model‑T) to a smaller student model (Model‑S). Distillation can be based on output probabilities, hidden‑layer representations, or attention distributions. Examples include Distilled BiLSTM (teacher BERT, student single‑layer BiLSTM) and MobileBERT, which narrows BERT via bottleneck layers and employs layer‑wise distillation with feature‑map transfer (FMT) and attention transfer (AT). MobileBERT also replaces layer‑norm with linear normalization and GeLU with ReLU to accelerate inference.

Structure‑Preserving Compression : Methods such as parameter sharing, low‑rank factorization, and attention decoupling reduce model size without altering architecture. Parameter sharing (as in ALBERT) and low‑rank factorization shrink storage but do not speed up inference, whereas attention decoupling removes redundant cross‑sentence attention in sentence‑pair tasks, improving speed.

The combined use of these techniques can achieve compression ratios up to 10× (or 40× when combined with quantization) while maintaining comparable performance on most NLP benchmarks, enabling BERT‑based models to run efficiently on mobile devices.

References :

[1] Ganesh P., Chen Y., Lou X., et al., “Compressing large‑scale transformer‑based models: A case study on BERT,” arXiv:2002.11985, 2020.

[2] Tang R., Lu Y., Liu L., et al., “Distilling task‑specific knowledge from BERT into simple neural networks,” arXiv:1903.12136, 2019.

[3] Sun Z., Yu H., Song X., et al., “MobileBERT: a compact task‑agnostic BERT for resource‑limited devices,” arXiv:2004.02984, 2020.