Large‑Scale Pretrained Model Compression and Distillation: AdaBERT, L2A, and Meta‑KD

Guest: Li Yaliang, PhD, Alibaba

Editor: Chen Dong, Southeast University

Platform: DataFunTalk

Overview: This presentation focuses on large‑scale pretrained model compression and distillation from an AutoML perspective, introducing three consecutive works from Alibaba DAMO Academy: AdaBERT, L2A, and Meta‑KD.

Background: Since the 2018 release of BERT, pretrained language models have become a dominant paradigm in NLP, achieving state‑of‑the‑art results on tasks such as text classification, named‑entity recognition, and machine reading comprehension. However, the parameter count of these models (e.g., 400 MB for a 12‑layer transformer, >1 GB for a 24‑layer version) creates two practical problems: limited deployment on resource‑constrained devices and slow inference speed.

Typical Compression Techniques:

Model distillation (e.g., DistilBERT, TinyBERT, MiniLM)

Quantization (e.g., Q‑BERT)

Parameter sharing (e.g., ALBERT)

Figure 2: Common research directions for BERT compression.

AdaBERT (Task‑Adaptive BERT Compression with Differentiable Neural Architecture Search):

Motivation: Conventional compression methods ignore downstream‑task specificity, leading to sub‑optimal performance when a single compressed model is fine‑tuned for many tasks. AdaBERT searches for a compact model that is specialized for a particular task.

Model Design: A CNN‑based search space is defined because CNN operations enjoy better hardware support and are well‑studied in NAS. The search objective balances knowledge retention (via a task‑specific loss) and efficiency (model size), resulting in a combined loss illustrated in Figure 5.

Figure 5: Knowledge‑efficiency trade‑off loss design.

Optimization: A differentiable NAS approach with Gumbel‑trick weighting is used to train a super‑net that contains all candidate operations, enabling efficient architecture search.

Experiments: AdaBERT achieves near‑optimal performance while drastically reducing parameters and providing noticeable inference speed‑up (see Figure 8).

Figure 8: Experimental results of AdaBERT.

L2A (Learning to Augment for Data‑Scarce Domain BERT Knowledge Distillation):

Motivation: AdaBERT assumes abundant task‑specific data, which is unrealistic in many domains. L2A addresses data scarcity by automatically generating useful augmented data during distillation.

Model Design: L2A combines transfer learning with an adversarial data generator. The generator creates synthetic source‑domain data that is fed to both teacher and student models; a reinforcement‑learning loop refines the generator based on distillation loss and evaluation metrics.

Figure 10: L2A model architecture.

Results: Across four downstream tasks, L2A consistently improves performance, especially when training data is limited; in some cases the compressed student outperforms the original large model (see Figures 12‑13).

Figure 12: L2A experimental evaluation.

Meta‑KD (Meta‑Knowledge Distillation Framework for Language Model Compression across Domains):

Motivation: Selecting appropriate source domains for L2A requires manual effort. Meta‑KD introduces a meta‑teacher that learns cross‑domain knowledge, enabling automatic adaptation to any target domain.

Two‑Stage Training:

Stage 1 – Meta‑teacher learning: a model is trained to aggregate knowledge from multiple source domains.

Stage 2 – Meta‑distillation: the student model learns from the meta‑teacher on the target domain, even when the target domain has never appeared in the source data.

Figure 14: Meta‑KD model architecture.

Experiments: On large benchmarks (MNLI, Amazon) covering multiple domains, Meta‑KD yields consistent performance gains; notably, it performs well even without any explicit domain data (see Figures 15‑17).

Figure 15: Meta‑KD results on MNLI.

Q&A Highlights:

Q: Why can a small model sometimes outperform a large one? A: Large models may contain task‑irrelevant knowledge that acts as noise; a compact model can discard this noise, leading to better performance.

Q: Are compression techniques applicable to generative tasks? A: Current work focuses on classification, intent detection, and information extraction; generative tasks have not been explored yet.

Thank you for attending.