Unlocking BERT: How Its Transformer Engine Powers State-of-the-Art Text Classification

1. Model Input / Output

The full name of the model is Bidirectional Encoder Representations from Transformer (BERT). Its goal is to learn rich semantic representations of text from large‑scale unlabeled corpora, which are then fine‑tuned for specific NLP tasks. The primary input consists of token (character or word) embeddings, optionally initialized with pretrained vectors such as Word2Vec. The output is a contextualized vector for each token that captures the meaning of the whole sentence.

In the current Chinese implementation, the input is the sum of three components: character vectors , a text vector learned automatically during training, and a position vector that distinguishes tokens appearing at different positions.

Text vector: automatically learned representation that encodes global semantic information.

Position vector: added to each token to reflect its position in the sequence.

2. Pre‑training Tasks

2.1 Masked LM

Randomly mask 15% of the tokens in a sentence and ask the model to predict them. For the masked tokens, 80% are replaced with the special token [MASK], 10% with a random token, and the remaining 10% are left unchanged. This forces the model to rely heavily on surrounding context.

2.2 Next Sentence Prediction

Given two sentences, the model predicts whether the second sentence follows the first in the original document. This is analogous to the paragraph‑reordering exercise used in language‑learning tests.

3. Model Structure

3.1 Attention Mechanism

Attention lets the network focus on the most relevant parts of the input. It uses three vectors: Query , Key , and Value . For a target token, the Query is compared with the Keys of all other tokens to obtain weights, which are then used to combine the corresponding Values into an enhanced representation.

3.2 Self‑Attention and Multi‑head Self‑Attention

Self‑Attention treats each token as a Query against all tokens (including itself). Multi‑head Self‑Attention runs several independent attention heads in parallel, allowing the model to capture different semantic aspects of the sequence and then linearly combines their outputs.

3.3 Transformer Encoder

A Transformer encoder layer consists of Multi‑head Self‑Attention followed by three key operations: a residual (skip) connection, layer normalization, and two linear transformations. Stacking these layers builds a deep encoder.

3.4 BERT Model

By stacking 12 or 24 Transformer encoder layers, BERT models with 110 M or 340 M parameters are obtained.

4. Text Classification Experiments

The model was evaluated on six Chinese and English datasets (product‑review sentiment, Sentiment_XS, stance detection, AG’s News, Yelp Review Full, Yahoo! Answers). Comparisons were made with XGBoost, char‑level CNN, attention‑based RNN, SVM, and other strong baselines.

Key results include:

Product‑review sentiment: BERT F1 scores 71 % (positive), 76 % (negative), 92 % (neutral), outperforming all baselines.

Sentiment_XS: BERT accuracy 90.01 %, higher than CNN (87.12 %).

Stance detection (theism): BERT F1 75.51 %, about 10 % above the previous best.

AG’s News: BERT accuracy 94.6 %, the highest among all methods.

Yelp Review Full and Yahoo! Answers: BERT achieved the second‑best accuracies (66.0 % and 74.2 %).

These results demonstrate BERT’s strong generalization across diverse text‑classification tasks and languages.

5. Conclusion

The article analyzed BERT’s internal architecture and pre‑training objectives, and showed that its contextualized representations lead to state‑of‑the‑art performance on many benchmark datasets. Future work will explore deeper (24‑layer) configurations and efficiency improvements.

Citations

Devlin J, Chang MW, Lee K, et al. BERT: Pre‑training of Deep Bidirectional Transformers for Language Understanding. arXiv preprint arXiv:1810.04805, 2018.

Vaswani A, Shazeer N, Parmar N, et al. Attention Is All You Need. Advances in Neural Information Processing Systems, 2017: 5998‑6008.

Zhang L, Chen C. Sentiment Classification with Convolutional Neural Networks: An Experimental Study on a Large‑scale Chinese Conversation Corpus. CIS 2016.

Mohammad S, Kiritchenko S, Sobhani P, et al. SemEval‑2016 Task 6: Detecting Stance in Tweets.

Zarrella G, Marsh A. MITRE at SemEval‑2016 Task 6: Transfer Learning for Stance Detection.

Wei W, Zhang X, Liu X, et al. pkudblab SemEval‑2016 Task 6: A Specific Convolutional Neural Network System for Effective Stance Detection.

Wei P, Mao W, Zeng D. A Target‑Guided Neural Memory Model for Stance Detection in Twitter.

Joulin A, Grave E, Bojanowski P, et al. Bag of Tricks for Efficient Text Classification.

Conneau A, Schwenk H, Barrault L, et al. Very Deep Convolutional Networks for Natural Language Processing.

Johnson R, Zhang T. Deep Pyramid Convolutional Neural Networks for Text Categorization.