dots.tts: Open‑Source Continuous Autoregressive TTS Model for Sustainable, Scalable Voice Synthesis

The Xiaohongshu dots team has open‑sourced dots.tts , a 2 billion‑parameter, fully continuous, end‑to‑end autoregressive speech synthesis model that serves as a sustainable, extensible TTS foundation.

Key Open‑Source Features

Zero‑shot voice cloning : achieves the best average content accuracy and speaker similarity on the Seed‑TTS‑Eval benchmark.

Six checkpoints : include pre‑training, self‑correcting alignment, 4‑step/2‑step/1‑step low‑step generation, and 1‑to‑1 audio‑text (1T1A) dual‑stream mode.

Interactive streaming support : native audio streaming and 1T1A mode allow text and audio streams to advance interleaved.

Low‑latency inference : under the paper’s single‑GPU 4‑step setting, the normal streaming mode has a first‑packet latency of 85.4 ms, while 1T1A achieves 54.4 ms.

Extensibility : dots.tts.edit builds on the same continuous autoregressive base to enable text, emotion, prosody, and pause editing.

Full training pipeline : training, inference, fine‑tuning, distillation, and dual‑stream inference code are released under Apache 2.0.

Evaluation Results

On Seed‑TTS‑Eval (three subsets), the SOAR version reports WER/CER of 0.94 %, 1.30 %, and 6.60 % with speaker similarity (SIM) of 81.0, 77.1, and 79.5 respectively. Averaged across subsets, SOAR attains 79.2 % SIM and 2.95 % WER/CER, surpassing all compared systems in both metrics.

On the MiniMax‑Speech 24‑language benchmark, the SOAR version reaches an average SIM of 83.9, ranking first in 19 languages and tying for first in two others. In the EmergentTTS‑Eval suite, the SOAR variant obtains the highest syntactic‑complexity score (65.7 %) and the Base version achieves the best open‑source emotion‑expressiveness score (72.7 %).

Why Zero‑Shot Cloning Reflects Base Capability

Autoregressive models generate the next acoustic segment conditioned on the target text, reference audio, and previously generated acoustic history. Zero‑shot cloning tests content modeling, reference‑audio understanding, speaker preservation, and long‑range stability simultaneously, revealing the balance of these abilities in dots.tts.

Continuous vs. Discrete Representations

Discrete tokenization quantizes continuous acoustic variation into a limited codebook, causing irreversible information loss. Continuous representations retain richer details, yielding higher PESQ‑NB (3.99–4.23 vs. 2.40–2.92) and SIM (up to 0.969 vs. 0.85 for discrete schemes). The VAE‑based continuous latent space preserves speaker identity more faithfully before autoregressive modeling.

AudioVAE and Semantic Encoder

To mitigate error accumulation in long‑range generation, dots.tts first trains an AudioVAE with two stages: (1) high‑quality waveform reconstruction and smooth latent space formation; (2) multi‑task supervision (WavLM frame‑level alignment, ASR, emotion, speaker ID) that injects semantic structure into the latent space.

The second stage yields a causal Semantic Encoder that extracts stable semantic history from generated audio and feeds it back to the autoregressive decoder, reducing drift in timbre, rhythm, and content.

Low‑Step Inference Techniques

1) Self‑correcting alignment (SOAR) generates off‑distribution states by rolling its own predictions forward, then learns to pull them back to clean targets without reward models.

2) MeanFlow distills the teacher’s trajectory into an average velocity, allowing a student to skip larger intervals and reduce forward passes per acoustic block.

3) sCM (simplified Consistency Model) constrains predictions at different time points to remain consistent, enabling stable 2‑step generation.

4) Reward‑Aware DMD adds a speech‑quality reward and a two‑time‑scale update (TTUR) to achieve high‑quality single‑step generation.

All four checkpoints (4‑step, 2‑step, 1‑step) are released for developers to trade off quality, throughput, and latency.

1‑to‑1 Audio‑Text Dual‑Stream Mode

In normal streaming, the full text prefix is fed first, then audio blocks are generated sequentially. In 1T1A mode, the upstream language model begins emitting tokens, and the TTS system consumes each new token to produce audio concurrently, achieving a first‑packet latency of 54.4 ms.

SGLang‑Omni integrates dots.tts and supports up to 16 concurrent inference streams on a single GPU, demonstrating both low‑latency real‑time interaction and high‑throughput batch serving.

Precise Voice Editing – dots.tts.edit

dots.tts.edit extends the base architecture to perform precise audio editing. It accepts source text, source audio, edit instructions, and target text as a unified context and generates the edited audio in a single pass. Supported operations include text replacement/insertion/deletion, emotion alteration, pitch‑rate adjustment, and pause manipulation, either singly or combined.

doteBench Evaluation Suite

The team also releases doteBench , a bilingual benchmark containing 2 081 cases (1 841 single‑edit, 240 multi‑edit). It measures instruction execution, unedited region preservation, and overall naturalness using WER/CER, emotion/prosody metrics, WDTW‑Dur, WDTW‑F0, and speaker similarity.

dots.tts.edit leads across five task categories in both instruction following and local preservation.

Overall Design Principles

The authors define dots.tts as a “base” model built on four pillars: (1) solid fundamental generation (zero‑shot cloning, content accuracy, cross‑language speaker consistency); (2) a technically extensible route (continuous representation, autoregressive modeling, semantic AudioVAE); (3) a single model supporting multiple deployment shapes (SOAR, low‑step distillation, 1T1A streaming); (4) a platform for future capabilities (dots.tts.edit).

By open‑sourcing six checkpoints, the full training pipeline, and the dual‑stream code, the project invites researchers to explore new representations and training methods, developers to adapt the model for industry scenarios, and product teams to integrate it into voice agents, creation tools, and full‑duplex dialogue systems.