Deploy Microsoft VibeVoice TTS for Real‑Time Multi‑Speaker Audio

Model Overview

VibeVoice is an open‑source family of text‑to‑speech (TTS) models released by Microsoft Research, designed for long‑dialogue scenarios. The flagship VibeVoice‑TTS‑1.5B can synthesize up to 90 minutes of continuous audio for up to four speakers within a 64K context window, addressing the typical drift in timbre and semantic breaks of conventional TTS systems. It supports English and Chinese generation as well as cross‑language voice conversion.

The model uses a dual acoustic‑semantic tokenizer architecture and runs at an ultra‑low 7.5 Hz frame rate, compressing 24 kHz raw audio by a factor of 3 200, which is about 80× more efficient than mainstream Encodec models.

Realtime Variant

VibeVoice‑Realtime‑0.5B adopts an interleaved‑window design, achieving a first‑packet latency of roughly 300 ms and enabling seamless "generate‑while‑play" streaming. With only 0.5 B parameters, it is lightweight enough for direct embedding in applications. While English is the primary language, the model also performs well in German, French, Italian, Japanese, Korean, Dutch, Polish, Portuguese, and Spanish.

The model can automatically detect semantic cues in the input text and produce matching emotional intonations such as anger, apology, or excitement, while preserving each speaker’s tone, rhythm, and timbre over long conversations.

Typical Use Cases

Generating natural multi‑speaker dialogue audio for podcasts or audiobooks without coordinating actors.

Real‑time voice assistants, live‑stream dubbing, and game NPC dialogue.

Assistive reading systems for the visually impaired, reducing listening fatigue compared with traditional screen readers.

Enterprise training audio and conversational customer‑service bots.

Licensing and Deployment

VibeVoice is released under the MIT license and can be deployed locally or in the cloud without subscription fees. The model is available as a pre‑built image on the 算网 GPU platform.

Deployment Steps on the 算网 GPU Platform

Open the official website (https://sumw.com.cn/) and navigate to the GPU marketplace.

Locate the community image for VibeVoice, select the desired version, and confirm the rental.

Start the instance and open JupyterLab.

In the terminal, execute the following commands:

cd /mnt/VibeVoice_mlu

source /path/to/your/env/bin/activate

export HF_ENDPOINT=https://hf-mirror.com

export PYTHONPATH=$PYTHONPATH:$(pwd)

echo "hello！" > test.txt

python run_mlu.py \

--model_path microsoft/VibeVoice-Realtime-0.5B \

--txt_path test.txt \

--speaker_name Emma \

--device mlu \

--output_dir ./output_tts_test

After the command finishes, the generated audio files will appear in the output_tts_test directory.