Open-Source ASR Optimization: Solving Misrecognition of Proper Nouns and Real-Time Lag

Why Open‑Source ASR Needs Careful Tuning

With options like Whisper, SenseVoice, and sherpa‑onnx proliferating, real‑world deployments often encounter two classic complaints: proper‑noun errors and inability to keep up with live speech. The article organizes mitigation methods into an executable decision tree and a layered mechanism to reduce trial‑and‑error.

Two Separate Bottlenecks

Accuracy and speed stem from different constraints. To improve accuracy, the author recommends a progression from audio quality, hot‑word / language‑model bias, switching to a stronger Chinese base model, and finally domain‑specific fine‑tuning. To improve speed, the path goes through smaller models, INT8 quantization, and choosing the right runtime (streaming sherpa‑onnx for real‑time, faster‑whisper for offline batch). The two tracks often pull in opposite directions; there is no "accurate, fast, and free" universal solution.

Five‑Layer Optimization Framework

1. Input – 16 kHz mono, denoise, VAD (zero cost, often overlooked).

2. Model – replace with a stronger base (e.g., SenseVoice, Qwen3‑ASR) or apply domain fine‑tuning.

3. Decoding – adjust beam size, add hot‑word bias, perform LM rescoring.

4. Deployment – ONNX + INT8, use sherpa‑onnx for streaming or faster‑whisper for GPU batch.

5. Post‑processing – punctuation model, ITN (inverse text normalization), custom dictionary replacement.

Six Accuracy‑Boosting Mechanisms (Easy → Hard)

Hot‑word / context bias – raise scores of domain‑specific terms; supported by FunASR, WeNet, sherpa‑onnx; no retraining needed.

LM rescoring – generate N candidates then re‑rank with N‑gram or neural LM; useful for homophones and grammar constraints; common in icefall and WeNet pipelines.

Prompt / context for large‑model ASR – prepend cues like "Speech transcription without text normalization:"; leverages LLM priors (Qwen3‑ASR, Fun‑ASR‑Nano).

VAD + segmentation – cut silence and invalid segments before decoding; FunASR includes FSMN‑VAD, asr_tool provides endpoint detection.

Audio front‑end processing – resample to 16 kHz mono, denoise (RNNoise, DeepFilterNet), AGC/volume normalization.

Domain fine‑tuning – train on your own "audio + annotations"; data < 10 h → hot‑words + LM, 10‑100 h → LoRA/adaptor, 100‑1000 h → encoder + adaptor, > 1000 h → full‑parameter fine‑tuning; yields the highest ROI.

Five Speed‑Optimization Paths

Model side – distillation (distil‑whisper) gives 6‑10× speedup with slight accuracy loss; INT8 quantization yields 2‑4× CPU speedup; switch to smaller architecture (Zipformer 14M vs 1.7B LLM).

Runtime – choose engine: sherpa‑onnx (CPU streaming, low latency), faster‑whisper (GPU batch for long audio), whisper.cpp (edge devices, no Python), vLLM (service‑grade deployment of Qwen3‑ASR).

Streaming architecture – true streaming pipeline: microphone → fixed chunk (e.g., 100 ms) → incremental decode → partial results → endpoint detection → final result → reset; smaller chunk reduces latency but raises CPU scheduling cost; 4‑8 threads is a typical sweet spot.

System‑level – GPU/NPU for offline batch, skip silence via VAD, parallel segmentation of long audio, reduce beam to 1 for 2‑3× speed gain.

Real‑time specific – true streaming is mandatory for live scenarios.

How to Verify Improvements

Use a fixed test set of 100‑500 representative samples and change only one variable per experiment (e.g., add hot‑words, swap model). Measure CER/WER (Chinese commonly uses CER), RTF (< 1 for real‑time), first‑word latency, and domain‑term accuracy.

Recommendation Ordering by ROI

Accuracy (easy → hard) : 1) audio standardization, 2) hot‑words / prompt, 3) adjust beam + add punctuation/ITN, 4) switch to stronger Chinese base, 5) LM rescoring, 6) domain fine‑tuning.

Speed (easy → hard) : 1) INT8 quantization + reduce beam, 2) switch to faster‑whisper / sherpa‑onnx, 3) adopt smaller or distilled model, 4) GPU + batch processing, 5) true streaming architecture.

Scenario‑Specific Combos

Desktop real‑time Chinese – hot‑words + better model; keep sherpa‑onnx INT8 streaming; default model streaming‑***‑zh‑14M INT8 for CPU.

Meeting transcription – SenseVoice / Qwen3‑ASR + punctuation model for accuracy; faster‑whisper GPU batch with segmented parallelism for speed.

Vertical domains (medical, legal, finance) – FunASR fine‑tuning + hot‑words + LM rescoring for accuracy; export ONNX to sherpa‑onnx for custom service.

Multilingual / dialect – Qwen3‑ASR + prompt for accuracy; vLLM service or 0.6B edge model for speed.

Key Takeaways

• Accuracy issues stem mainly from audio quality, missing hot‑words, weak language model, and domain mismatch; speed issues stem from model size, lack of quantization, and unsuitable runtime.

• Follow the five‑layer framework: improve input, model, decoding, deployment, and post‑processing.

• Prioritize the KPI (CER vs. RTF) before optimizing.

• Validate each change with a stable test set and appropriate metrics.