Building a Reproducible, Scalable ASR Evaluation Framework for 2025‑2026

Why Build a Unified ASR Evaluation Framework?

ASR now spans large models, streaming, and multilingual scenarios. Systems like Qwen3‑ASR, FireRedASR2S, and Fun‑ASR report impressive numbers, but inconsistent evaluation conditions (different datasets, normalization rules, decoding parameters) can mislead model selection.

Key issues highlighted:

On AISHELL‑1, the presence or absence of punctuation normalization can change CER by more than 0.5 percentage points.

Batch decoding versus chunked streaming yields incomparable RTF and WER values.

Cloud API calls and local ONNX inference follow different Dockerized pipelines, requiring unified containerization for batch runs.

"If you can't measure it, you can't improve it." – SpeechIO / SpeechColab philosophy

2025‑2026 Evaluation Trends

The Hugging Face Open ASR Leaderboard (github.com/huggingface/open_asr_leaderboard) now supports three tracks: short English, long English, and multilingual speech. According to arXiv:2510.06961, the platform compares over 60 models across more than ten public datasets and standardizes both WER and the inverse real‑time factor RTFx (audio duration ÷ inference time, larger is faster).

Accuracy: Conformer encoder combined with Transformer/LLM decoder achieves the best WER.

Efficiency: CTC/TDT decoders deliver high RTFx, suitable for long audio and batch processing.

Fairness: The same decoding hyper‑parameters are used for a given model across all datasets.

Hardware: Benchmarks are run on NVIDIA A100‑80GB; GPU model must be recorded.

For Chinese real‑world scenarios, SpeechIO maintains a test set covering 46 sub‑scenes (news, live streams, podcasts, dialect movies, lyrics, hearing‑impaired speech). In the January 2025 ranking, cloud APIs achieve an overall CER between 2.99 % and 10.10 %.

LLM‑driven ASR robustness is evaluated on noise, far‑field, dialect, singing, and code‑switching conditions. FireRedASR2S reports an average CER of 9.67 % across 24 public datasets.

Reference Architecture: TestSet Zoo · Model Zoo · Benchmark Pipeline

① TestSet Zoo (datasets/*): Combines academic corpora with the SpeechIO real‑scenario collection.

② Model Zoo (models/*): Provides a unified directory layout for cloud APIs and local models.

③ Benchmark Pipeline: Data preparation → recognition → text normalization → scoring → ranking.

TestSet Layering Recommendations

L1 regression set: AISHELL‑1 test, LibriSpeech test‑clean (fast CI).

L2 extended set: AISHELL‑2, WenetSpeech, GigaSpeech, Common Voice.

L3 scenario set: SpeechIO ZH00001‑46 (mandatory for Chinese product selection).

L4 robustness set: noise, far‑field, dialect, singing (selected per business needs).

Model Integration: model‑image Specification

Each model resides in a self‑contained directory containing a Dockerfile, model.yaml, SBI inference entry, and assets. model.yaml must declare language (zh/en) and sample_rate (typically 16000).

Standardized Five‑Stage Pipeline

Stage 1‑2 – Data Preparation & Batch Recognition: generate_test_data.py creates wav.scp and trans.txt. Run ./SBI wav.scp output_dir for batch inference; logs are written to log.SBI.

Stage 3 – Text Normalization (Critical): Chinese: textnorm_zh.py (uppercase, half‑width, filler removal, optional erhua removal). English: textnorm_en.py or Whisper‑style TN used by Open ASR Leaderboard. Without consistent normalization, benchmark scores become artificially high or low.

Stage 4‑5 – Scoring & Ranking: Chinese is scored by CER, English by WER. Results are written to DETAILS.txt and RESULTS.txt. Record model version, git hash, GPU type, batch size, and evaluation date.

Deep Evaluation with NeMo ASR Evaluator

ENGINE: Supports offline, chunked, and offline‑by‑chunked inference plus noise/silence augmentation.

ANALYST: Computes WER/CER, insertion/deletion/substitution counts, and buckets metadata by duration, emotion, etc.

exist_pred_manifest: Allows skipping inference and recomputing metrics from existing predictions.

Six‑Step Implementation Checklist

Adopt SpeechColab as the base, set LEADERBOARD environment variables, and launch quickly.

Download the L1 dataset and run a smoke test.

Package the first model following the model‑image specification.

Execute benchmark stages 1‑3 and verify RESULTS.txt.

Commit the text‑normalization configuration to Git; enable nightly CI for the L1 set.

For streaming or robustness evaluation, overlay NeMo or Open ASR Leaderboard scripts on top of the pipeline.

Common Pitfalls & Rigor Principles

Do not use different beam sizes for different models when aiming for a fair comparison.

Do not mix academic‑set numbers with SpeechIO figures in a single table without a dataset column.

Report Paraformer offline CER 1.94 % versus online CER 3.34 % on AISHELL‑1 separately.

Conclusion

The evaluation core consists of a TestSet, a Model, and a standardized Pipeline.

2026 trends point to multi‑track evaluation, dual metrics (WER + RTFx), and dedicated robustness subsets.

Chinese evaluation must include SpeechIO; English and multilingual evaluation aligns with the Open ASR Leaderboard tracks.

Consistent text normalization is a prerequisite for comparability and must be version‑controlled.

External numbers should always cite dataset, normalization rules, hardware, model version, and evaluation date.