DataFunTalk

Oct 18, 2022 · Artificial Intelligence

Large‑Model and Small‑Model Interaction: Knowledge Distillation and Reverse Distillation Techniques

This article explains how large‑scale NLP models can be paired with smaller models through task‑related and task‑unrelated knowledge distillation, progressive multi‑stage distillation, and reverse distillation, thereby reducing training costs, accelerating inference, and even allowing small models to improve large‑model training via sample‑value assessment.