Fine‑Tune a Chinese BERT Model for Cloze Tasks in 30 Minutes

Preface

Learning NLP is like leveling up in a game; each stage is a hurdle to overcome.

Level 1 – Understand NLP concepts and boundaries.

Level 2 – Use an existing model.

Level 3 – Fine‑tune a model for your own business.

Level 4 – Define a brand‑new model.

Previously we shared an article on quickly using an NLP model; this piece is a small progression.

Exploring level 3

The article takes about 30 minutes and aims to review NLP concepts and fine‑tune a Chinese BERT model for a cloze task.

NLP Introduction

Development History

NLP tasks have two clear stages: the pre‑BERT era of basic neural networks and the post‑BERT era (Bertology).

Reference: https://zhuanlan.zhihu.com/p/148007742

1950‑1970 – Rule‑based methods.

1970‑early 2000s – Statistical methods.

2008‑2018 – Introduction of deep learning (RNN, LSTM, GRU).

Present – 2017 Transformer architecture, 2018 BERT released, achieving state‑of‑the‑art results on 11 GLUE tasks.

BERT Family

Current Research Directions

Two main directions:

Natural Language Understanding (NLU)

Natural Language Generation (NLG)

Reference: https://zhuanlan.zhihu.com/p/56802149

Below is the NLP task taxonomy from HuggingFace.

Essential NLP Concept: Neural Networks

Neurons

This article highlights two key points for a high‑level understanding.

Neuron

A single neuron is the basic unit, analogous to a biological neuron where dendrites receive inputs and the axon sends the output.

Mathematical form:

Output = f(∑(x·w) + θ)

Each neuron accepts multiple inputs (x₁…xₙ), each multiplied by a weight (w₁…wₙ), summed, added to a bias (θ), passed through an activation function f to produce the output.

Activation functions add non‑linearity, enabling the network to model complex relationships.

Weights and bias are learned during training; the training process adjusts them to minimize prediction error.

Neural Network Workflow

Loss function: measures error between prediction and target. Back‑propagation: propagates error to update weights. Learning rate: step size controlling weight updates. Optimizer: algorithm that iteratively finds suitable weights.

In practice, libraries like PyTorch provide ready‑made loss functions and optimizers, and most scenarios use pre‑trained models out‑of‑the‑box.

Pre‑trained Models

BERT is a pre‑trained model; we briefly review the concept.

Reference: https://mp.weixin.qq.com/s?__biz=MzkxNTIwMzU5OQ==∣=2247492139&idx=1&sn=81edc7c73cbe7bf3462ae56d02171cf3

What Is a Pre‑trained Model?

Third‑party institutions release models trained on massive datasets that can be used directly.

Training cost illustration:

How to Use Pre‑trained Models

Most are hosted on HuggingFace; Baidu Paddle is a domestic alternative.

Domestic platform note: Baidu Paddle exists but HuggingFace has broader adoption.

Two usage patterns on HuggingFace:

1️⃣ Use the ready‑made pipeline with a single line of code.

2️⃣ Use the low‑level Transformers API (model, tokenizer, etc.).

Low‑level API steps:

Tokenization – split sentences into tokens and map to vectors.

Prediction – run the model inference.

Decoding – map output vectors back to words to form a sentence.

Advantages and Limitations

Advantages:

Engineering: plug‑and‑play, saves training cost and time.

Strong generalization from massive pre‑training data.

Limitations:

Pre‑trained models may not capture domain‑specific nuances, acting like a versatile but not specialized tool.

Solution: fine‑tune the model on custom data.

Fine‑tuning adapts a pre‑trained model to a specific business scenario, improving performance on targeted tasks.

Fine‑tuning BERT

We fine‑tune a Chinese BERT model on a cloze (masked language modeling) task.

Few Chinese cloze fine‑tuning examples are available, making this a novel demonstration.

What Is BERT?

BERT is trained by predicting masked tokens, which yields excellent sentence‑level semantic understanding.

Masking strategy: 80% replace with [MASK] , 10% replace with a random word, 10% keep unchanged.

Example:

Original: 我爱中国

Masked: 我爱[MASK]国

Normal Result

Fine‑tuning Goal

Goal: make the model predict the fictional historical figure “诸葛涛”, demonstrating custom knowledge injection.

Fine‑tuning Procedure

Online notebook: https://colab.research.google.com/drive/12SCpFa4gtgufiJ4JepLMuItjkWb6yfck?usp=sharing

Step 1: Prepare Custom Corpus

train.json example:

Code to load the corpus:

Step 2: Define Trainer

Define training and test sets:

Step 3: Train Model

Training code:

Training log:

Training completed:

Verification Results

Successfully added “诸葛涛” to the model’s predictions.

Conclusion

After completing this tutorial, we have successfully explored level 3: reviewed NLP fundamentals and fine‑tuned a Chinese BERT model for a cloze task.

Pre‑trained models are a boon for ordinary users; fine‑tuning lets anyone build a domain‑specific NLP model, turning every practitioner into a tuning engineer.

Further Reading

HuggingFace course:

https://huggingface.co/course/chapter7/3?fw=pt

https://huggingface.co/course/en/chapter5/5?fw=pt

Book: “Practical NLP with BERT”.