KuaiSearch's PERKS Pre‑trained Language Model Sets New Record on the CLUE Benchmark

The KuaiSearch search technology team recently refreshed the CLUE 1.1 language classification leaderboard with a score of 80.618, approaching the human annotation benchmark of 84.100 and marking a new industry breakthrough for pre‑trained language models.

Pre‑trained language models are the foundation of modern NLP. Following the pioneering BERT from Google and OpenAI's GPT‑3, many Chinese‑focused variants have emerged. KuaiSearch's model, PERKS (Pre‑trained Embedding Representation for Kuai Search), is built on real search business scenarios and benefits from extensive practical experience.

CLUE is one of the most authoritative Chinese language understanding benchmarks, covering text classification, semantic understanding, and more. KuaiSearch's top performance is on the classification sub‑task, which requires fine‑grained semantic discrimination of subtle phrase variations.

PERKS was trained on a massive, high‑quality Chinese corpus assembled from terabytes of video‑related text and external open‑source data, cleaned and filtered to reflect both KuaiSearch community and external characteristics.

Multi‑stage, multi‑task training methodology

The team designed a three‑stage training pipeline:

Stage 1 – Basic semantic knowledge learning : The model learns Chinese grammar and basic semantics through a dynamic whole‑word masking prediction task.

Stage 2 – General community semantic learning : Token‑level tasks (dynamic whole‑word masking, character shuffling, knowledge prediction) and sentence‑level tasks (sentence distance, order, and source prediction) are used to capture both internal and external semantic cues.

Stage 3 – Task‑specific semantic learning : The model adapts to downstream tasks, e.g., using contrastive learning as the main objective for a dual‑tower semantic model, with auxiliary tasks to fine‑tune performance.

Model architecture optimizations

Various architectural tweaks—such as Pre‑LayerNorm, a hybrid of relative and absolute positional encodings—were validated and incorporated to improve performance across different downstream scenarios like video content understanding and query analysis.

Engineering optimizations

To handle terabyte‑scale data efficiently, PERKS abstracts a distributed DataSet for training samples and employs Ring All‑Reduce, FP16, recompute, LAMB optimizer, and gradient accumulation as standard techniques. The system is also prepared for future integration of BM25 and ANN negative‑sample generation.

A demo video shows PERKS outperforming BERT in top‑5 predictions, correctly identifying both in‑domain items (e.g., “small gift”) and out‑of‑domain entities (e.g., “small hall”).

Deployed at large scale in KuaiSearch, PERKS has improved content regularization (+14.08%), query analysis (50% Bad Case resolution), semantic recall (Recall@5 +4%), and semantic re‑ranking (+30%). The CLUE challenge helped the team assess external knowledge learning and guide further model refinements for diverse business scenarios.

About the KuaiSearch team

The team focuses on enhancing user experience in video search, researching video content processing, pre‑trained models, semantic relevance, authority, timeliness, click‑through modeling, and recommendation, continuously bridging cutting‑edge technology with real user needs.