How Unified Prompt Tuning Boosts Few-Shot NLP Performance Across Tasks

Background

As pretrained language models grow to billions or trillions of parameters, their performance on natural language understanding improves, but fine‑tuning them on downstream tasks typically requires large labeled datasets. In many vertical domains, only a few annotated examples are available, limiting accuracy. Prompt‑tuning‑based few‑shot learning can leverage knowledge from pretraining, yet limited training data still constrains performance. UPT aims to further improve few‑shot accuracy by exploiting cross‑task data.

Algorithm Architecture

Unified Prompt Tuning (UPT) extends existing few‑shot methods by unifying downstream and pretraining tasks into a Prompt‑Options‑Verbalizer (POV) format, allowing a single model to learn a general prompting strategy. The task construction is illustrated below:

UPT can transform single‑sentence classification, sentence‑pair matching, and self‑supervised pretraining tasks into this unified paradigm, combining classic few‑shot advantages with meta‑learning to reduce over‑fitting.

Unified Prompting Paradigm

Each NLP task is expressed as:

P (Prompt) : task‑specific template containing at least one [MASK] token.

O (Option) : a set of candidate label words presented as a question.

V (Verbalizer) : mapping between label words and actual class labels.

Examples:

Sentiment classification: "[X]. Is great or bad? It was [MASK]."

Paragraph coherence: "[X1]. Is this paragraph the same as the next: [X2]? It was [MASK]."

Incorporating Self‑Supervised Tasks

During pretraining, UPT augments the standard Masked Language Modeling (MLM) task with a Prompt‑MLM auxiliary task, without retraining the underlying language model. Prompt‑MLM selects high‑frequency adjectives, clusters them by semantic similarity, and creates a binary classification prompt where the mask corresponds to an adjective and the option is a dissimilar adjective from another cluster.

Evaluation

Experiments used RoBERTa‑large as the backbone, fine‑tuning each downstream task with only 16 labeled examples per class. Accuracy was compared against standard fine‑tuning and few‑shot baselines (LM‑BFF, PET, P‑tuning, PPT) on nine public datasets. Results show that UPT consistently outperforms the baselines.

Additional experiments on SuperGLUE benchmarks confirm the superiority of UPT, and the PAI team achieved first place on the FewCLUE Chinese few‑shot leaderboard, surpassing major industry competitors.

The UPT implementation will be contributed to the open‑source EasyNLP framework ( https://github.com/alibaba/EasyNLP ), inviting NLP researchers and practitioners to adopt the method.

References

Jianing Wang, Chengyu Wang, Fuli Luo, et al. "Towards Unified Prompt Tuning for Few‑shot Text Classification." EMNLP Findings 2022.

Chengyu Wang, Minghui Qiu, Taolin Zhang, et al. "EasyNLP: A Comprehensive and Easy‑to‑use Toolkit for Natural Language Processing." EMNLP 2022 (accepted).

Tianyu Gao, Adam Fisch, Danqi Chen. "Making Pre‑trained Language Models Better Few‑shot Learners." ACL/IJCNLP 2021.

Timo Schick, Hinrich Schütze. "Exploiting Cloze‑Questions for Few‑Shot Text Classification and Natural Language Inference." EACL 2021.

Timo Schick, Hinrich Schütze. "It's Not Just Size That Matters: Small Language Models Are Also Few‑Shot Learners." NAACL‑HLT 2021.

Xiao Liu, Yanan Zheng, Zhengxiao Du, et al. "GPT Understands, Too." CoRR 2021.

Chengyu Wang, Jianing Wang, Minghui Qiu, et al. "TransPrompt: Towards an Automatic Transferable Prompting Framework for Few‑shot Text Classification." EMNLP 2021.