FireRedTTS3: Zero‑Shot Multilingual Voice Cloning, Design, and Editing in One Model

The FireRed team released FireRedTTS3, a next‑generation speech generation and editing model that simultaneously supports three core abilities—zero‑sample multilingual voice cloning, voice design via natural‑language description, and pinpoint‑accurate speech editing—using a single unified architecture.

Recent advances in TTS have made zero‑sample voice cloning mature, but user demands have shifted from merely replicating a voice to controlling it: describing desired timbre in words and editing specific segments without affecting the rest. Achieving this control requires a representation that encodes both semantic content (style, intent) and fine‑grained acoustic details.

Existing technical routes each have drawbacks: non‑autoregressive flow‑matching lacks strong instruction‑following; discrete token quantization loses acoustic detail; continuous autoregressive models suffer from error accumulation that leads to drift in timbre and prosody.

FireRedTTS3 addresses the error‑accumulation problem at the representation level with RedAE, a continuous speech tokenizer trained with a frozen “semantic teacher” (FireRedAudio) that already knows content semantics and speaker characteristics. During RedAE training, the teacher’s encoder features are aligned to the tokenizer output via an MSE (semantic distillation) loss, after which the teacher is discarded, keeping inference cost unchanged. RedAE also omits KL regularization to preserve acoustic fidelity.

On top of RedAE, the model adopts a lightweight LLM‑DiT generation framework consisting of three components: (1) an Aggregator that down‑samples RedAE outputs; (2) a Backbone Transformer initialized from Qwen‑3, inheriting strong text understanding and instruction‑following; and (3) a DiT module that denoises latent patches conditioned on noisy inputs and historical clean latents. Two variants are provided: FireRedTTS3‑Base, focused on multilingual cloning (backbone from Qwen‑3‑Base, speaker embeddings via CAM++, language tags), and FireRedTTS3‑Instruct, which unifies cloning, design, and editing (backbone from Qwen‑3‑Instruct, planning‑then‑synthesis pipeline that converts user commands into a 12‑dimensional acoustic plan before synthesis).

Extensive evaluation shows FireRedTTS3 leading on four public test sets. On Seed‑TTS‑Eval (zero‑sample cloning), the Base model achieves the lowest average error rate (3.04 %) and highest speaker similarity (78.8 %). On MiniMax‑MLS‑Test (24 languages), it records the lowest average error rate (3.75 %) and highest similarity (84.8 %). InstructTTSEval (voice design) sees the Instruct model attaining top scores across acoustic‑parameter, style‑description, and role‑play tasks, thanks to its plan‑then‑synthesize design. Finally, on Ming‑Freeform‑Audio‑Edit (speech editing), FireRedTTS3‑Instruct attains leading WER, NoEdit‑WER, ACC, SIM, RDE, and RAE metrics, confirming precise local edits without affecting untouched regions.

In summary, by embedding semantics directly into the speech representation and using a simple yet powerful LLM‑DiT generation pipeline, FireRedTTS3 simultaneously solves cloning, design, and editing with a single model, sets new performance records across multiple benchmarks, and paves the way for future expansion of languages, dialects, and richer instruction control.