How Bilibili’s IndexTTS2 Achieves Real‑Time, Emotion‑Rich Voice Translation

01 | Background of Original‑Voice Video Translation

We present a new capability that converts Chinese video speech into foreign‑language original‑voice style dubbing, preserving the speaker’s tone, rhythm, and personality, thus delivering a natural, immersive multilingual experience.

The motivation stems from the growing demand for authentic, in‑scene multilingual content that goes beyond mere comprehension to retain emotional nuance and speaker identity.

Key challenges include loss of voice personality, cognitive load from subtitles, and high localization costs.

02 | Perception‑Consistent Speech Generation Modeling

Traditional TTS focuses on naturalness and intelligibility but lacks multi‑dimensional modeling required for video‑level voice translation, which we define as perception‑consistent reconstruction across three dimensions: speaker identity, acoustic space, and multi‑source time‑frequency structure.

Speaker identity reconstruction: Our IndexTTS2 clones the original speaker’s timbre using minimal reference audio, achieving high‑fidelity voice personality preservation.

Acoustic space preservation: The system retains reverberation, microphone distance, and background noise cues to maintain spatial hearing consistency.

Multi‑source timbre fusion: Human voice, background music, and ambient sounds are jointly modeled to avoid perceptual breaks.

2.1 Integrated Solution for Cross‑Language Voice Consistency, Emotion Transfer, and Speed Control

We address three technical hurdles: voice‑style consistency across languages, quantifiable emotion transfer, and natural speech‑rate adaptation to match original video timing.

Voice‑style gaps arise when Chinese tonal characteristics shift to a harsher English timbre.

Emotion transfer is difficult because linguistic cues for affect differ between languages.

Speech‑rate control must handle up to 30% length differences while preserving the original audio’s temporal constraints.

2.2 Solving Multi‑Speaker Confusion

Accurate speaker diarization is essential for multi‑speaker videos. We propose a fine‑grained semantic segmentation followed by speaker clustering, enhanced low‑frequency speaker detection, and an upgraded end‑to‑end speaker model that robustly distinguishes voices even in noisy, overlapping conditions.

03 | Cross‑Language Semantic and Cultural Adaptation

Beyond translation accuracy, the system must balance speech rhythm, contextual style, and cultural terminology.

Dynamic rhythm‑density balancing: Adjust text length based on target language density while respecting original audio duration.

Contextual and stylistic consistency: Model speaker identity, dialogue structure, and domain register to maintain coherent expression.

Precise handling of proper nouns and cultural references: Use a dynamic term‑bank and context‑aware mapping to preserve meaning and emotional impact.

3.1 RIVAL: Adversarial Reinforcement Learning Framework

We introduce RIVAL, a min‑max game between a reward model and a large language model, integrating qualitative and quantitative preferences (speech length, translation metrics, and creator style) to jointly optimize rhythm control, translation quality, and style adaptation.

3.2 Deep Search for Proper‑Noun and Cultural Adaptation

Deep Search combines query generation, real‑time web retrieval, and summarization to obtain accurate translations for domain‑specific terms, especially in anime and gaming contexts.

04 | Audio‑Visual Alignment for Video Reconstruction

After achieving perceptual audio consistency, we address temporal and spatial alignment between audio and visual streams.

Semantic‑visual decoupling of subtitle regions: A multimodal model locates and erases original subtitles, preventing language and visual mismatch.

Audio‑driven lip‑sync generation: A diffusion‑based generator, guided by a 3D VAE and reference network, produces high‑fidelity lip movements synchronized with the translated speech while preserving the speaker’s identity.

4.1 Subtitle Removal

We employ a hybrid architecture that fuses multimodal large‑model understanding with traditional OCR to precisely locate subtitle regions and apply cross‑frame smoothing for consistent erasure.

4.2 Lip‑Sync Alignment

The pipeline inputs a lower‑face mask, reference video, and audio, using a 3D VAE for temporal modeling and a reference network for identity preservation, achieving robust lip‑sync even under large head angles and occlusions.

05 | Conclusion

Original‑voice translation bridges language barriers while retaining speaker personality, emotional nuance, and cultural context, reducing localization costs and enhancing global content accessibility. Future work will expand language support, open‑source IndexTTS2, and invite collaboration from AI researchers and creators.