How We Replaced BERT with a Lightweight TextCNN to Slash GPU Costs

Background

Among recent natural language processing models, BERT is the most representative due to its excellent performance, but its large parameter size consumes excessive computational resources during inference and hinders model iteration in production environments.

Zuoyebang processes massive daily text data that must be classified with NLP models. The labeling workflow spans multiple phases, each with different thresholds, inter‑phase dependencies, and limited supervised data, requiring fine‑tuning BERT for every phase.

Challenges

1) New task requests demand fresh BERT fine‑tuning. 2) As task count grows, the number of deployed BERT instances rises, consuming substantial GPU resources and complicating scheduling.

The goal is to find a solution that delivers inference quality comparable to BERT while using far fewer computational resources and offering strong scalability.

Feasible Replacement Strategies

Lightweight techniques considered include:

Knowledge Distillation : a teacher‑student training framework where a compact student model learns from a pretrained teacher model.

Pruning : removing less important connections without changing the model architecture.

Quantization : converting high‑precision parameters to lower‑precision types to reduce model size and compute time.

The desired new model should have a small parameter count for fast iteration and maintain good performance on text‑classification tasks.

Chosen Solution

We combined knowledge distillation (teacher‑student) with active learning. BERT predictions were used to filter TextCNN inference results; the loss and probability distribution of TextCNN guided data augmentation. The iterative workflow is illustrated in the accompanying diagram.

Training Data

Zuoyebang’s data exhibits:

Temporal periodicity.

Repeated expression patterns.

Competitive statements (e.g., “求表扬” vs. “表扬单上没有我”).

Complex emojis.

Noise from ASR/OCR.

Data processing steps:

Collect long‑range data covering multiple cycles.

Apply sampling strategies:

Deduplicate redundant symbols and filter noisy sentences.

Split training and test sets with temporal isolation to evaluate generalization.

Training Method

The iterative process:

Train a base TextCNN model.

Evaluate TextCNN against predefined metrics.

If metrics are met, test the model on recent data; otherwise, analyze loss and probability outputs to adjust sampling for the next round.

Repeat until the model satisfies precision, recall, and throughput requirements.

Effect Evaluation

Metrics were computed on a test set using BERT inference as the gold standard. Results across three iterations are summarized below:

Iteration 1 – Precision: 0.5435, Recall: 0.405, F1: 0.464. Iteration 2 – Precision: 0.912, Recall: 0.438, F1: 0.591. Iteration 3 – Precision: 0.876, Recall: 0.882, F1: 0.878.

Inference speed was measured on CPU for TextCNN and on GPU/CPU for BERT. TextCNN achieved significantly higher throughput (up to 1449.6 items/s on 20‑core CPU) compared with BERT@GPU (83 items/s).

Model Deployment

To handle multiple models simultaneously, a task‑queue system was employed:

Assign priorities to models.

Conduct stress tests to determine optimal concurrency.

Deployment diagrams illustrate the architecture and resource allocation.

Technical Summary and Outlook

The scenario features abundant historical data, numerous BERT models consuming GPU resources, and tolerance for modest performance loss in the replacement model. By leveraging teacher‑student distillation and active learning, the TextCNN model meets business requirements, frees valuable GPU resources, shortens processing time, and scales easily to additional classification tasks.