RVCBench at NeurIPS 2026: Benchmarking 18 Voice Cloning Models for Real-World Robustness

Background: Voice Cloning Leaves the Lab

Voice cloning (zero-shot TTS) aims to synthesize arbitrary speech matching a target speaker from only a few seconds of reference audio. Recent autoregressive codec-token language models and diffusion/flow-matching models achieve impressive similarity and naturalness on clean test sets, enabling applications like personalized assistants and dubbing. However, real-world deployment introduces distribution shifts: noisy or multi-speaker reference recordings, anomalous or adversarial text (e.g., scam scripts), long-form or cross-lingual generation, downstream compression (MP3, AAC, Opus) and narrowband telephony, and active anti-cloning perturbations added by speakers to protect their voices. Existing evaluations focus on clean-condition quality, lacking systematic robustness characterization.

RVCBench Design: Four Dimensions, Ten Task Categories

RVCBench structures robustness along the full cloning pipeline into four dimensions (Figure 1):

Input Robustness : RVC-AudioShift varies speaker attributes across 12 English accents and demographic groups; RVC-TextShift replaces standard text with hallucination-prone anomalies (typos, format irregularities, code-mixed segments) and scam robocall scripts.

Generation Robustness : RVC-LongContext tests long reference audio and minute-level text; RVC-Expression evaluates emotional/style transfer; RVC-Multilingual covers monolingual English (VCTK, LibriTTS), Chinese (AISHELL-1), French (Common Voice), and English-to-Chinese cross-lingual cloning.

Output Robustness : RVC-Compression simulates AAC/MP3/Opus at various bitrates and narrowband telephony; RVC-Detectability measures separability from real speech using nine deepfake detectors (self-supervised, GNN, CNN, audio LLM judges).

Audio Perturbation Robustness : RVC-PassiveNoise adds café, station, train noise and multi-speaker interference; RVC-AdvNoise applies active protection perturbations from SafeSpeech and variants SPEC, Enkidu, POP; RVC-AntiProtect tests whether DEMUCS denoising can bypass protections.

The benchmark comprises 18 evaluations across 10 task categories, 14,370 test samples, 225 speaker entries (204 unique speakers after deduplication).

Evaluation Setup: Unified Metrics and 18 Representative Models

RVCBench adopts a unified metric suite (Figure 3): Word Error Rate (WER), Mel-Cepstral Distortion (MCD), Speaker Similarity (SIM), Speaker Verification Acceptance (SVA), Mean Opinion Score (MOS), Short-Time Objective Intelligibility (STOI), and emotional alignment scored by Gemini-3-Flash (0–3).

The main paper evaluates 18 open-source models spanning three paradigms:

Autoregressive codec-token models (6) : FishSpeech, XTTS-v2, Spark-TTS, IndexTTS, MOSS-TTSD, Higgs.

Diffusion/flow-matching non-autoregressive models (8) : MaskGCT, StyleTTS2, F5-TTS, OZSpeech, OpenVoice, PlayDiffusion, VibeVoice, ZipVoice.

Hybrid LM + diffusion/flow models (4) : CosyVoice, Qwen3-TTS, MGM-Omni, GLM-TTS.

Appendix adds Fish Audio S2, MOSS-TTS v1.5, dots.tts, Higgs TTS 3.

Key Findings

Input Shifts: Text Anomalies Primarily Hurt Content Accuracy

Replacing standard VCTK text with hallucination-style text raises WER across models, some severely (Figure 3a), indicating abnormal formats and rare tokens break text-speech alignment. Scam scripts cause smaller but visible WER increases (Figure 3b). On the reference audio side, accent affects cloning quality: Indian-accented references yield highest average MCD, Canadian the lowest, though some accent groups contain only one speaker, so results signal potential gaps rather than definitive rankings.

Generation Tasks: Reference Length, Long Text, and Emotion Expose Weaknesses

Reference length significantly impacts quality. As reference duration grows from a few seconds to 8–12 seconds, MCD drops and SIM rises rapidly; gains saturate beyond that (Figure 4a,b). Sensitivity varies: CosyVoice, Higgs, MGM-Omni continue benefiting from longer references, while StyleTTS2 and OZSpeech show limited SIM improvement even with long references.

Long text is a universal challenge. Switching from LibriTTS to minute-level LibriSpeech long text increases WER and MCD for most models (Figure 5), with content omissions, substitutions, and acoustic drift accumulating over length. No model is fully immune.

Emotional alignment drops sharply on scam texts. On normal text nearly all models score near 3/3; on scam robocall scripts scores fall noticeably and inter-model variance widens (Figure 6). Scam texts carry strong, explicit emotions (urgency, fear, pressure, reassurance) that models fail to reliably convert into prosody and speaking style.

Multilingual settings reveal large disparities. FishSpeech WER rises from 0.03 (English VCTK) to 0.47 (Chinese AISHELL-1); Higgs from 0.19 to 0.78. Hybrid models (CosyVoice, MGM-Omni, GLM-TTS) maintain higher speaker similarity in Chinese and cross-lingual tasks.

Output Side: Compression Causes Consistent Degradation; Detectability Varies by Model

All models exhibit MCD increase and STOI decrease after compression. Narrowband telephony harms spectral consistency most; low-bitrate codec impact on intelligibility differs per model (Figure 4c,d), showing output robustness depends on architecture and vocoder design, not merely clean-generation quality.

Detectability: OpenVoice, OZSpeech, PlayDiffusion are easier to distinguish from real speech; CosyVoice, Spark-TTS, MGM-Omni, Qwen3-TTS, GLM-TTS show lower separability under standard detectors. The paper treats detectability as a complementary safety signal—generation quality and detectability must be assessed separately.

Audio Perturbations: Noise Degrades Cloning; Active Protection Resists Simple Denoising

Adding natural background noise degrades all 18 models on average: WER relative worsening reaches 265.6%, MCD, SIM, SVA, MOS drop 19–36% (Figure 7a). Sensitivity varies widely (Figure 7b): OpenVoice MCD slightly improves, ZipVoice changes little, while Spark-TTS and MOSS-TTSD suffer largest MCD increases. About half the models show non-significant WER change; the other half degrade. Multi-speaker interference universally reduces identity, naturalness, and spectral consistency as interference strength grows.

Active anti-cloning protection is relatively effective (Figure 8a). Averaged over 18 main models, clean-reference SIM ≈ 0.492; SPEC and SafeSpeech reduce it to ≈ 0.223 and 0.242 (>50% drop). Protection strength varies: POP nearly disables FishSpeech (SIM 0.01), while Qwen3-TTS retains highest SIM across protections. MOSS-TTSD and Spark-TTS are vulnerable to both noise and adversarial perturbations.

RVC-AntiProtect applies SPEC protection then DEMUCS denoising (DNS64) to attempt removal (Figure 8b–d). Denoising reduces spectral distortion for some models, but generated speech remains notably worse than clean-reference; SIM and MOS drops persist for most models. Naive denoising does not bypass current active protections.

Three Deeper Observations

Clean-condition high scores ≠ robustness. A model may excel on clean VCTK (e.g., StyleTTS2, PlayDiffusion WER 0.01) yet have low speaker similarity (0.24, 0.43) versus CosyVoice (0.58) and GLM-TTS (0.57). Single-metric leaderboards overestimate real-world reliability.

Larger models are not necessarily more robust. No monotonic robustness gain with parameter count: 639M CosyVoice outperforms 5.97B Higgs across language shifts; 2.16B GLM-TTS beats Higgs on Chinese and cross-lingual tests. Autoregressive models compete on clean English but degrade under language shift; diffusion/flow models excel at content accuracy in simple English but lag in speaker similarity; hybrids balance multilingual and cross-lingual settings. Architecture family, conditioning mechanism, multilingual training, and speaker representation design explain robustness differences more than scale. (Analysis is descriptive, not causal, due to differing training data and objectives.)

Robustness and security are two sides of the same coin. Users want stability under noise/compression; speakers want perturbations to block unauthorized cloning. RVCBench evaluates both in one protocol, enabling joint discussion of “how stable is the model?” and “how strong is the protection?”.

From Paper Benchmark to Open-Source Toolkit

RVCBench v2 (GitHub main branch) is refactored as a pip-installable evaluation toolkit ( pip install "rvcbench[eval]", PyPI v2.2.1). After rvcbench setup-scorers downloads metric models, it offers two workflows:

Score your own audio : Use metrics.Evaluator Python API to compute SIM, SVA, WER, SpeechMOS, MCD, STOI, emotional consistency in batch.

Evaluate with RVCBench data (Figure 9): rvcbench prompts --suite core-v1 --output prompts/ exports reference audio and text; generate with your model (any environment/API) saving as <id>.wav; then

rvcbench score --suite core-v1 --generated outputs/my_model --output results/my_model --device cuda

produces submission.json with per-task scores, coverage, confidence intervals. rvcbench compare or multiple --generated dirs yield comparison.md tables. Prompts provided in ZipVoice and Seed-TTS batch-list formats compatible with F5-TTS, CosyVoice native scripts.

Three suites: onboarding-v1 (52 utterances, smoke test), core-v1 (480 utterances, covers 16 of 18 paper evaluations), full-v1 (12,724 utterances, larger scale; protection tasks only in core-v1). --tasks filters scenarios (e.g., Chinese, background noise); --resume continues interrupted runs. Repository includes 32 model adapters, 5 protection methods, and a denoising stage for protection/countermeasure research. Note: current suites are preview; paper results use a separate reproduction protocol on v1 branch; deepfake detectability and LLM emotional judging not yet packaged. Full docs at https://nanboy-ronan.github.io/RVCBench/docs/.

Conclusion

Voice cloning is moving into the real world, which never provides clean test sets. RVCBench’s value lies not in a single ranking but in dissecting capability boundaries: where cloning remains reliable, where it fails, and how far active protection can deter misuse. The work argues future systems must treat robustness as a first-class design goal, covering noisy/multi-speaker references, out-of-distribution text, cross-lingual and long-form generation, emotional prosody transfer, compression artifacts, and perturbation-resistant speaker representations. The benchmark and toolkit aim to help researchers uncover weaknesses early and provide a reproducible, extensible foundation for speech generation safety research.