Efficient Deployment of Speech AI Models on GPUs

Introduction – The article introduces the motivation for deploying speech recognition (ASR) and speech synthesis (TTS) pipelines on GPUs, highlighting latency, concurrency, and cost issues in cloud deployments and the need for efficient GPU utilization.

GPU‑Based ASR Solution – NVIDIA and the WeNet community built an ASR pipeline on Triton Inference Server, using Fbank feature extraction, Conformer/U2++ encoders, and CTC prefix beam‑search decoding. Challenges such as module ordering, pipeline parallelism, custom pre‑/post‑processing, conditional logic, and GPU utilization for both streaming and non‑streaming workloads are addressed with Triton’s business‑logic scripting, dynamic batching, and custom backends.

Decoding Optimizations – CUDA implementations for CTC prefix beam‑search and TLG decoding move the entire decoding loop onto the GPU, eliminating CPU‑GPU copy overhead and achieving >10× speed‑up over CPU‑based decoders. The trade‑off between speed and support for language‑model rescoring is discussed.

Whisper Acceleration with TensorRT‑LLM – Large‑scale Whisper models are accelerated using TensorRT‑LLM. Optimizations include fused MHA kernels with Flash Attention, specialized kernels for layer‑norm and matrix ops, KV‑Cache handling, and INT8 quantization (weights in INT8, compute in FP16). These improvements yield ~40% faster inference on V100 compared to Faster‑Whisper while maintaining lower CER.

GPU‑Based TTS Solution – A streaming TTS pipeline combines Incremental FastPitch (chunk‑wise processing with causal convolutions and masked multi‑head attention) and a stream‑GAN discriminator to ensure smooth chunk concatenation. Triton custom C++ backends provide inflight batching and zero‑code model ensembles for encoder, vocoder, and blending modules, all accelerated with TensorRT.

Voice Cloning – Multi‑speaker FastPitch is trained on 220 speakers from AIShell‑3, CSS‑10, and LJSpeech. Fine‑tuning uses 20 user‑recorded utterances, initializing speaker embeddings from the most similar training speaker. Triton servers host both cloning and TTS models, enabling batched fine‑tuning and inference via Python backends.

Future Plans – Continued Whisper optimization (TensorRT‑LLM + Triton), code‑switched ASR training, broader TTS acceleration for SOTA models, multilingual TTS, and LLM‑driven TTS are outlined.

Q&A Highlights – Answers cover CER‑Hypo usage for pseudo‑label filtering, TensorRT support on consumer GPUs, upcoming quantization libraries, dynamic batching dependencies, Whisper streaming feasibility, open‑source status of TensorRT‑LLM, and its performance trade‑offs.