Understanding XLNet: Differences from BERT, Innovations, and Experimental Analysis

The article, authored by Zhang Junlin, a senior algorithm expert at Sina Weibo AI Lab, provides an in‑depth review of XLNet, a recent NLP model that has attracted significant attention for outperforming BERT on several benchmarks.

It first explains the two main families of language models: autoregressive models such as GPT and ELMO, which predict the next token from left‑to‑right (or right‑to‑left) context, and autoencoder (denoising) models like BERT, which mask tokens and predict them using bidirectional context, outlining each approach’s strengths and limitations.

XLNet’s core contribution is to combine the advantages of both families by introducing the Permutation Language Model (PLM). Through a clever use of attention masks and a dual‑stream attention mechanism, XLNet allows each token to attend to both preceding and succeeding tokens without explicit [MASK] symbols, preserving a left‑to‑right training format while effectively incorporating bidirectional information.

The model also incorporates two additional innovations: (1) the Transformer‑XL architecture, providing relative positional encoding and segment‑level recurrence to handle long documents, and (2) a dramatically larger pre‑training corpus that adds Giga5, ClueWeb, and Common Crawl data to the original BooksCorpus and Wikipedia, following the scaling strategy of GPT‑2.

Experimental comparisons on reading‑comprehension datasets (RACE, SQuAD 2.0), the GLUE benchmark, and other NLP tasks demonstrate that XLNet consistently surpasses BERT, with especially large gains on long‑document tasks. The analysis attributes these improvements to the new PLM objective, the long‑context handling of Transformer‑XL, and the increased data volume, while also noting that the data‑scale factor contributes roughly 30 % of the overall gain.

The author concludes that XLNet’s design opens new research directions, offering clear benefits for generation‑oriented tasks (e.g., summarization, translation) and for applications requiring long‑range context, and predicts rapid adoption of XLNet‑based models in these areas.

Author bio: Zhang Junlin is a director of the China Computer Federation, holds a Ph.D. from the Institute of Software, Chinese Academy of Sciences, and has held senior technical positions at Alibaba, Baidu, and Yonyou before joining Sina Weibo AI Lab.