Baidu Semantic Computing: ERNIE, SimNet, and Future Directions in Natural Language Processing

Author Sun Yu, Baidu NLP chief architect, presents a report originally delivered at the 2019 Natural Language Processing Frontier Forum, organized into three parts: semantic representation, semantic matching, and future work.

Semantic computing at Baidu began with efforts to represent, analyze, and compute the meaning of human language, encompassing semantic representation, matching, analysis, and multimodal computation.

The report introduces Baidu's progress in semantic representation, highlighting the ERNIE (Enhanced Representation through Knowledge Integration) series. Early work (2007) used search‑based methods to build term‑level representations, later moving to topic models and large‑scale word embeddings. From 2013 onward, Baidu invested heavily in distributed training of massive word‑embedding models (1 T tokens, 1 M vocabulary) and hierarchical softmax techniques.

With the rise of deep learning, Baidu adopted Transformer‑based pre‑training. Experiments on five Chinese benchmarks (lexical analysis, NER, inference, QA, sentiment, similarity) showed ERNIE 1.0 consistently outperformed BERT, CoVe, GPT, and ELMo. English evaluations on GLUE and SQuAD 1.1 confirmed the gains, and qualitative analyses on cloze tasks demonstrated the benefit of knowledge‑enhanced masking.

SimNet, Baidu's neural semantic matching framework introduced in 2013, combines DNN‑based representation with pairwise training. It has evolved through three stages—seed (BOW model), development (CNN/RNN with knowledge infusion), and expansion (integration of authority, click, and QA signals). SimNet is deployed across search, news recommendation, advertising, and dialogue platforms.

Recent advances include the SimNet‑QC‑MM model, query‑document joint modeling with matching matrices, and heterogeneous GPU/CPU inference architectures. Model compression techniques (quantization, hashing) have reduced embedding size from 32 bits to 4 bits, cutting memory consumption by 87.5% and benefiting both server‑side search and mobile applications.

Future work will focus on universal semantic representations, leveraging prior knowledge, multilingual and multimodal embeddings, and domain‑specific applications such as medical and legal NLP.

Resources: Baidu's open‑source ERNIE repository ( http://github.com/PaddlePaddle/ERNIE ), ERNIE 2.0 pre‑trained models, and the RLTM paper ( https://arxiv.org/abs/1906.09404 ).

---END---