Xiaohongshu's dots.tts.edit V1.1 Tops Open-Source Speech Editing Benchmarks
Xiaohongshu releases dots.tts.edit V1.1, an end-to-end precise speech editing model adding local style description editing and significantly improving prosody and emotion control, achieving top open-source results on doteBench and SpeechEditBench while open-sourcing model weights, training data pipeline, and evaluation suite.
Xiaohongshu has upgraded its end-to-end precise speech editing model to dots.tts.edit V1.1, targeting the need to modify specific words, prosody, emotion, or pauses in a recording while keeping the rest of the audio stable and natural.
Core Capabilities
The model supports bilingual (Chinese/English) and multi-speaker speech editing. Users select words in the transcript and specify text changes or acoustic parameters (speech rate, pitch, pause, or free-form natural-language style descriptions). Multiple edits can be applied in a single generation pass. A demonstrated example edits a mixed Chinese-English, two-speaker utterance: replacing "here" with "Shanghai Jiao Tong University X-LANCE Lab" and correcting the English text, while preserving both speakers' voice characteristics and achieving natural transitions.
Model Architecture
dots.tts.edit builds on the dots.tts continuous autoregressive architecture, generating speech in the continuous latent space of an AudioVAE. The model takes the source audio, transcript, and edit instructions as input and produces the full edited output. Because of the autoregressive design, it requires no forced alignment between speech and text and no separate duration predictor; generation ends via an EOS token. All edit types share a single model.
Semi-Formalized Edit Instructions
To specify "what to change" and "how to change it," the project uses semi-formalized instructions aligned with the transcript. Operations are bound to exact word spans or boundaries, supporting parameters such as speed ratio, pitch shift, or free-form natural-language descriptions. This design ensures reliable target localization while keeping a flexible, extensible interface. A locally deployable speech editing application (https://dots-studio-dots-tts-edit.hf.space/) visualizes instructions as highlighted spans and operation tags, allowing users to select segments, set parameters, combine operations, and preview results.
Training Data Pipeline
Creating paired data for local editing is challenging: natural recordings rarely contain identical utterances with only the desired change, while independently synthesizing before/after segments introduces unwanted speaker or rhythm shifts, and naive splicing causes artifacts. The open-sourced data construction pipeline (https://github.com/studio-dots-ai/dots.tts.edit-data-pipeline) generates samples around explicit edit operations, maximally reusing surrounding audio, repairing local transitions, and filtering for both edit accuracy and naturalness. Multiple edit types are trained jointly, mixed with standard TTS data to preserve synthesis ability.
Evaluation Benchmark: doteBench
Existing speech editing benchmarks insufficiently assess local modifications. doteBench (https://github.com/studio-dots-ai/doteBench) evaluates three dimensions simultaneously:
Instruction Following: whether target text, emotion, speech rate, pitch, and pauses change as required.
Local Preservation: checks content, duration, pitch, and speaker identity in unedited regions.
Audio Quality: overall listening experience, clarity, and edit-boundary smoothness.
Metrics are normalized to a fixed scale; higher is better. On doteBench's five editing scenarios, dots.tts.edit V1.1 leads open-source models (AuK, Ming-UniAudio, Step-Audio-EditX) in both instruction following and local preservation, with audio quality on par.
Sequential Editing Results
Real-world editing often involves multiple rounds (e.g., change script, then adjust emotion, then add a pause). doteBench supports sequential evaluation where each step uses the previous output as input. In this setting, dots.tts.edit achieves a full-sequence success rate of 42.92% , more than 2× the best open-source baseline ( 18.33% ) (see paper Table 6, Sequential All column).
Third-Party Benchmark: SpeechEditBench
On SpeechEditBench, V1.1 attains the best joint success rates among open-source systems for text editing ( 94.00% ), emotion editing ( 15.07% ), and prosody editing ( 91.50% ). Notably, its prosody editing score also surpasses the closed-source systems compared in the SpeechEditBench paper.
Subjective Listening Tests
Human preference evaluations confirm higher average scores for instruction adherence and local preservation versus baseline models. Full test setup and results are in the paper (https://arxiv.org/abs/2608.02673).
Open-Source Suite
The release includes:
Demo Page: https://dots-studio-dots-tts-edit-demo.static.hf.space/
Interactive Playground: https://dots-studio-dots-tts-edit.hf.space/
Model Weights: https://huggingface.co/dots-studio/dots.tts.edit
Code: https://github.com/studio-dots-ai/dots.tts
Data Construction Pipeline: https://github.com/studio-dots-ai/dots.tts.edit-data-pipeline
doteBench Benchmark: https://github.com/studio-dots-ai/doteBench
Paper: https://arxiv.org/abs/2608.02673
Creators can audition and edit recordings directly; developers can deploy the model, integrate the application, and reuse the data pipeline and evaluation tools.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
