Designing a Telephone-Specific Chinese ASR: Base Model + RL Post-Training Wins

Why Generic ASR Fails on Phone Calls

Phone channels and microphones operate in fundamentally different acoustic worlds. The telephone channel's effective bandwidth is only 300–3400 Hz, while critical consonant distinctions (s/sh, z/zh, c/ch) reside in higher frequencies. Generic models trained on 16 kHz wideband audio must extrapolate when high-frequency content is entirely missing, causing error rates to spike.

Upsampling 8 kHz recordings to 16 kHz does not recover lost information — the spectrum above 3400 Hz remains empty. The only correct approach is training on real narrowband distributions.

Beyond bandwidth, phone calls introduce codec distortions (G.711, G.729, AMR compression, packet loss, echo, crosstalk), heavy accents and fast speech, dense digit sequences (amounts, dates, phone numbers, verification codes) where a single error is a business incident, and dual-speaker interactions with frequent overlap, backchannels, and interruptions — completely unlike read-speech data.

Paradigm Shift: Base Model + Post-Training

In 2026, ASR development no longer starts from scratch. Leading open-source speech foundation models have been pre-trained on tens of millions of hours of real speech, providing robustness to dialects, noise, and accents out of the box. Training a 1–2B parameter model from scratch would require million-hour labeled datasets and massive compute, yet likely underperform existing bases.

Three principles define the new paradigm:

Pluggable base: Open-source speech LLMs iterate every six months; architecture must allow swapping the base without rebuilding the system.

Domain adaptation as post-training: The problem shifts from "collect massive data and train from zero" to "adapt with small domain-specific data via post-training."

Moat migration: Competitive advantage moves from raw data hours to domain data uniqueness + iteration speed .

The model weights themselves are the most replaceable component. What truly belongs to you are four assets: the data engine, the post-training recipe, context engineering, and layered evaluation sets.

Overall Architecture: Five-Layer System

The base layer occupies only one layer. The "telephone-specific" differentiation lives in the context layer and alignment layer — the biggest increments over traditional approaches in 2026.

Context Layer: ASR That "Looks Up References"

Every phone call carries dynamic context: CRM records provide caller name, product, order history; outbound script state machines know the current offer. Retrieving industry hotwords and injecting them together with business context into the decoder instantly boosts hotword hit rates — no retraining, on-demand, a structural advantage of LLM-based decoders over traditional acoustic models.

Alignment Layer: Generative Error Correction + RL Post-Training

The base model first produces N-best candidates. A small LLM then takes candidates and context for a second-pass generative error correction (GER). Reinforcement learning further aligns digit strings, punctuation, and inverse text normalization (ITN) output formats directly to business specifications. After these two steps, the model outputs not "roughly correct characters" but transcripts ready for quality-inspection systems.

Data Engine: The Real Moat

Publicly available Chinese 8 kHz real-phone annotated data barely exists. This is both bad news and good news — whoever spins up the data engine first builds an uncopyable moat.

Four data streams feed the mixer:

Real phone recordings: Dual-channel, sentence-level annotated, de-identified and authorized. 100 hours suffice to kick off SFT.

Pseudo-labeling flywheel: Base model labels existing recordings → confidence bucketing → low-confidence + sampled high-confidence sent for human correction.

Channel simulation: 16 kHz public data passed through a standardized telephony chain — bandpass filtering → G.711 encoding → packet loss injection → noise addition — turning wideband data into realistic phone audio.

TTS/VC targeted synthesis: Hotword positive/negative examples, digit-heavy sentences, voice conversion for accent transfer.

Starting mix ratio reference: real : simulated : synthetic ≈ 5 : 3 : 2, dynamically adjusted based on layered evaluation results.

Training Recipe: SFT → RL → Distillation → QAT

Stage 2 (RL) Is the Biggest Gain: Verifiable Rewards

Phone scenarios offer a natural RL advantage: rewards are automatically computable and non-gameable. Character error rate, entity F1 for amounts/dates/phone numbers, and hallucination penalties (long text output during silence) are hard, verifiable metrics. One GRPO round yields digit recognition and output-format improvements that SFT alone cannot deliver.

Stage 4 (QAT) Pre-Empts "Post-Fine-Tuning Quantization Drop"

CPU private deployment requires quantization (int8/int4). Traditional workflows quantize after fine-tuning, often losing significant accuracy. Embedding quantization-aware training (QAT) into the recipe lets the model "know" it will be quantized during training, preventing deployment-time regressions.

Two Telephone-Specific Architecture Decisions

Decision 1: Role Labels Into the Decoder

Instead of running dual-channel audio through the model twice (left channel then right channel), speaker-role tags (agent/customer) are fed directly into the decoder. A single forward pass produces the full conversation with role labels. Overlapping speech is handled, and language priors cross channels: when the agent says "Let me verify your phone number," the customer channel's digit recognition automatically gains contextual boost.

Decision 2: Unified Streaming and Offline Model

New-generation bases prove unified streaming accuracy loss can be made negligible. Streaming capability is obtained via distillation, not by training a separate streaming model. This eliminates maintaining two model branches.

Training supervision signals are readily available: dual-track transcripts + simulated mixing naturally provide role labels.

Evaluation & Milestones: Acceptance Based on Entity F1, Not Raw CER

Phone business value lives in digit strings. Evaluation sets are layered and frozen from day one (industry × role × accent × channel). Acceptance metrics are entity F1, hallucination rate, and first-package latency — not a single raw CER. The layered evaluation set holds veto power over every iteration, blocking not only bad models but also "overall good, key scenarios weak" false progress.

Summary

No training from scratch: pluggable base, upgrade = swap adapter and re-run one round.

"Telephone-specific" = data distribution + post-training, not special network architecture.

Data engine is the moat: real phone + pseudo-labeling + channel simulation + targeted synthesis.

RL with verifiable rewards (CER / entity F1 / hallucination penalty) is the 2026 biggest increment.

Evaluation watches entity F1 and hallucination rate; layered eval sets have veto power.