Application of Large-Scale Pretrained Models in Alibaba Machine Translation

The rapid rise of large‑scale pretrained models such as ELMo, GPT, BERT, and ERNIE has dramatically advanced many NLP sub‑fields, yet their impact on machine translation (MT) and natural language generation (NLG) remains limited due to exposure bias and the mismatch between training and inference.

To bridge this gap, the authors propose the APT (Adaptive Pretraining‑Translation) framework, which dynamically fuses pretrained embeddings with MT embeddings on the encoder side and employs knowledge distillation on the decoder side to align the probability distributions of the two models.

The framework includes two key mechanisms: (1) a hierarchical attention‑based dynamic fusion that adjusts the proportion of pretrained and MT embeddings across layers, and (2) a knowledge‑distillation component that transfers token‑level and sentence‑level knowledge from the pretrained model to the MT model.

In addition, a novel Global Representation Extraction Transformer (GRET) is introduced, leveraging capsule networks and layer‑wise recurrent structures to capture global sentence‑level information that standard self‑attention mechanisms miss.

Extensive experiments on public WMT datasets (e.g., Chinese‑English, English‑German) demonstrate that APT improves three major MT tasks by 1.9–2.3 BLEU points, with BERT‑based encoders yielding the best results, while GRET reduces model parameters by 75% with comparable performance to Transformer‑Big.

Future work will explore multimodal extensions, multilingual pretraining for low‑resource languages, and applying the methods to document‑level and dialogue translation scenarios.