An Overview of NVIDIA NeMo for Speech AI: ASR Training, Chinese Support, and Related Applications

NeMo is NVIDIA's deep‑learning toolkit for conversational AI that supports automatic speech recognition (ASR), natural language processing (NLP), and text‑to‑speech (TTS), enabling end‑to‑end training and inference for a variety of speech‑AI tasks.

The speech‑AI pipeline starts with an audio input that passes through ASR (feature extraction, acoustic model, language model, decoder) to produce text, which is then processed by NLU for downstream tasks such as translation or query matching; the entire pipeline can be built within NeMo.

ASR has evolved from traditional HMM‑GMM models to modern end‑to‑end neural architectures like Wav2letter, DeepSpeech, LAS, Citrinet, and Conformer, with toolkits such as Kaldi, OpenSeq2Seq, EspNet, WeNet, and NeMo providing support.

NeMo supports a wide range of ASR model families—including LSTM, Jasper, QuartzNet, Citrinet, Conformer, and Squeezeformer—and offers both CTC and RNNT decoders, language‑model fusion, and streaming training/decoding configurations.

Training an ASR model with NeMo involves preparing a JSON manifest, extracting Mel‑spectrogram or MFCC features, applying augmentations like SpecAug, choosing a tokenization unit (character, subword, BPE), configuring the encoder (e.g., ConformerEncoder with customizable layers and dimensions), and running the provided training script; evaluation uses CER metrics via the speech_to_text_eval.py script.

For deployment, NeMo models can be exported to ONNX and served with NVIDIA Riva, which leverages TensorRT and Triton for accelerated inference and supports streaming decoding.

Chinese speech support includes Aishell‑1/2 preprocessing scripts, pretrained Citrinet‑CTC and Conformer‑Transducer models, and WFST‑based text normalization; TTS support for Chinese is planned for future releases.

Additional speech‑AI functionalities covered are voice‑activity detection (VAD) using the lightweight MarbleNet model, speaker diarization with TitaNet embeddings, clustering, and multi‑scale diarization, all provided as open‑source pretrained resources.

The article concludes with a Q&A addressing multi‑GPU training, Chinese TTS roadmap, vocabulary size considerations, MarbleNet resource usage, combined VAD‑SD pipelines, model sizes, inference chunk‑size settings, and other practical concerns.