Large-Scale Pretrained Model Compression and Distillation: AdaBERT, L2A, and Meta‑KD

The talk, presented by Dr. Li Yaliang from Alibaba and organized by DataFunTalk, focuses on the compression and distillation of large‑scale pretrained models from an automated machine‑learning perspective.

Background: Since BERT's 2018 release, massive pretrained models have dramatically improved NLP tasks but their size (hundreds of megabytes to gigabytes) hinders deployment on resource‑constrained devices, motivating model compression techniques such as distillation, quantization, and parameter sharing.

AdaBERT: AdaBERT addresses the limitation of task‑agnostic compression by using neural‑architecture search (NAS) to automatically design a small CNN‑based student model tailored to a specific downstream task. The search space is defined with multiple CNN operations, and the loss balances knowledge retention (task loss) with efficiency (model size). Experiments show AdaBERT achieves near‑optimal performance with the smallest parameter count and notable inference speed‑up.

L2A (Learning to Augment): L2A tackles data‑scarce domains by automatically generating augmented data via a generator trained with reinforcement learning. The generator produces useful synthetic examples for both source and target domains, enabling effective knowledge distillation even when labeled data are limited. Experiments across four tasks demonstrate consistent performance gains, especially when training data are scarce.

Meta‑KD: Meta‑KD introduces a meta‑teacher model that learns cross‑domain knowledge, allowing a student model to acquire task‑specific knowledge without manually selecting source domains. Training proceeds in two stages: meta‑teacher learning and meta‑distillation. Results on large datasets (MNLI, Amazon) across multiple domains show performance improvements, even when no explicit domain data are provided.

Q&A Highlights: The audience asked about why small models sometimes outperform larger ones and whether compression can be applied to generative tasks. The presenters suggested that large models may contain task‑irrelevant noise and that current work focuses on classification‑type NLP tasks.

Overall, the three works illustrate a progression from task‑adaptive architecture search (AdaBERT) to data‑augmentation‑driven distillation (L2A) and finally to meta‑learning‑based cross‑domain distillation (Meta‑KD), offering practical solutions for deploying efficient language models.