Generative Pretrained Structured Transformers: Unsupervised Syntactic Language Models at Scale (GPST) – Overview and Live Presentation

In the era of massive data, unsupervised pre‑training models are driving NLP forward. At ACL 2024 in Bangkok, the paper “Generative Pretrained Structured Transformers: Unsupervised Syntactic Language Models at Scale” was accepted.

A live online session “Paper Show Live #3” will be held on 14 August 2024, 18:30‑19:30, featuring Ant Group Research Institute associate researcher Hu Xiang, who will present the GPST model.

The GPST model introduces a generative syntactic language model that can be pre‑trained without manually annotated parse trees, scaling to 10 B tokens by using a log‑N‑complexity compositional language model (R2D2) and a “understand‑then‑memorise” training paradigm.

Experimental results show GPST outperforms GPT‑2 on text understanding, summarisation, and syntactic generalisation tasks, while training speed is more than 50 × faster than previous unsupervised syntactic models.

Key highlights: (1) novel unsupervised pre‑training GPST breaking the dependence on annotated data; (2) highly efficient training and superior performance across downstream tasks.

Live‑viewing details: 14 Aug 2024, 18:30‑19:30 on WeChat Channels (AntTech), Bilibili, and Ant Group Research Institute’s video channel. Please reserve your spot.