OmniVoice: A Zero‑Shot TTS Paradigm Covering 600+ Languages

OmniVoice, released by the k2‑fsa team (including Daniel Povey), targets the omnilingual zero‑shot TTS problem: producing speech for low‑resource languages and dialects without any speaker‑specific fine‑tuning, covering more than 600 languages.

Problem statement : Existing zero‑shot TTS models can clone voice from a few seconds of reference audio but typically support only dozens of languages, leaving many low‑resource languages underserved. The goal is to push speech synthesis to a truly global scale rather than achieving another English/Chinese SOTA.

Traditional two‑stage bottleneck : Most non‑autoregressive (NAR) pipelines follow a “text → semantic token → acoustic token” cascade. While this modularity eases training, errors in the semantic stage cannot be recovered in the acoustic stage, and low‑bitrate semantic representations lose fine details.

OmniVoice’s design choices :

Eliminate the intermediate semantic layer; predict multi‑codebook acoustic tokens directly from text and an optional prompt.

Use a bidirectional Transformer combined with a discrete diffusion objective, requiring only one training pass and one inference pass.

Incorporate the reference audio prefix as a prompt segment that participates in masked reconstruction together with the target segment.

Core architecture : The model receives a token sequence Y (instruction + text) and a prompt acoustic prefix X_prompt. The target acoustic matrix X_target is partially masked with a special [M] token. A bidirectional Transformer processes Y and the visible tokens, and a C‑head predicts the masked acoustic tokens. Training employs full‑codebook random masking, where each (time, codebook) position is independently Bernoulli‑sampled, resulting in roughly 50 % of positions contributing to the loss—significantly more supervision than layer‑wise masking schemes.

Key innovations :

Full‑codebook random masking (C‑times more effective supervision than layer‑wise masking).

LLM weight initialization: the backbone is initialized from a pretrained autoregressive large language model (e.g., Qwen‑3‑0.6B), preserving linguistic priors while using bidirectional attention at inference.

Capabilities and usage : OmniVoice supports three generation modes—Voice Cloning (reference audio + optional text), Voice Design (natural‑language description of gender, age, accent, pitch, etc.), and Auto Voice (no reference or instruction). Fine‑grained control includes non‑linguistic symbols ([laughter], [sigh]), Chinese pinyin with tone markers, and CMU phonemes for English. Inference can be run with a single pip install and one‑line commands (omnivoice‑demo, omnivoice‑infer, omnivoice‑infer‑batch).

Performance metrics : On a multilingual benchmark covering up to 102 languages, OmniVoice achieves SOTA intelligibility, speaker similarity, and naturalness. Community evaluations report a Chinese WER of ~0.84 % and a real‑time factor (RTF) of 0.025 on NVIDIA GPUs. Compared with CosyVoice, Qwen3‑TTS, F5‑TTS, Flow Matching, fish‑speech, and commercial ElevenLabs APIs, OmniVoice offers broader language coverage, a diffusion‑style token predictor, and open‑source accessibility without per‑character billing.

Training pipeline and ecosystem : The repository provides end‑to‑end training scripts (data preparation, multi‑GPU Accelerate training, fine‑tuning, evaluation). The default backbone is configurable (Qwen‑3‑0.6B) with flexible attention backends (flex_attention for PyTorch ≥ 2.5 or SDPA). Model weights and a Space demo are hosted on HuggingFace (k2‑fsa/OmniVoice). Community projects include omnivoice‑server (OpenAI‑compatible API) and OmniVoiceTTS (Docker image).

Conclusion : OmniVoice combines a single‑stage discrete NAR diffusion model with full‑codebook random masking and LLM initialization, delivering zero‑shot cloning for 600+ languages, voice design, fine‑grained control, an RTF of 0.025, and a permissive Apache‑2.0 license, making it a practical solution for large‑scale multilingual TTS deployment.