VoiceStudio: Local-First Open-Source Voice Platform for Cloning, Dubbing & TTS

What Is VoiceStudio?

VoiceStudio is an open-source, fully local AI voice workbench that covers the entire pipeline from voice cloning to video dubbing and audiobook production. It is not a single-purpose tool but a "local voice platform" with core capabilities:

Voice Cloning: Zero-shot cloning from a 3-second clean audio sample to synthesize arbitrary text.

Voice Design: Create voices from descriptions (gender, age, pitch, accent, style) without any sample.

Video Dubbing: Multi-language translation dubbing that preserves speaker identity and outputs subtitled video.

Audiobooks/Stories: Multi-character scripts, EPUB/PDF import, chapter rendering, export to .m4b.

Dictation & Transcription: System-wide hotkey for real-time transcription with speaker diarization.

Additional features include vocal isolation (separating vocals from background music), batch queues, and a model catalog for visual management of TTS/ASR models.

Local-First: Core Design Philosophy

The fundamental difference from cloud services like ElevenLabs is the data path. Cloud services upload audio and text to remote servers; VoiceStudio defaults to local-only, with remote access only via explicit opt-in. Telemetry is disabled by default and, even if enabled, sends only allow-listed usage metadata without content, filenames, or projects. The desktop app communicates with a local backend on localhost:3900 via loopback (no API key); remote access requires a shared PIN or API key.

16 TTS Engines + 11 ASR Engines

VoiceStudio is a multi-engine platform, not tied to a single model:

16 TTS engines: Default is k2-fsa's OmniVoice (600+ languages, clonable, controllable), plus CosyVoice 3, GPT-SoVITS, VoxCPM2, MLX-Audio (Apple Silicon), PocketTTS, Sherpa-ONNX, covering cloud-grade quality to CPU real-time lightweight.

11 ASR engines: WhisperX (default, word-level timestamps), Faster-Whisper, MLX Whisper, Parakeet, Moonshine, FunASR, adapted to different platforms.

All engines are managed visually in the model catalog: install/uninstall, select active engine, see device (CUDA/MPS/CPU/ROCm). Hardware recommendations from project docs:

Apple Silicon (M1–M4) → MLX-Audio / OmniVoice + MLX Whisper

NVIDIA GPU (8GB+ VRAM) → OmniVoice / CosyVoice 3 + WhisperX

Low VRAM / CPU only → PocketTTS + Faster-Whisper

Voice Cloning: Zero-Shot, 3 Seconds

Voice cloning uses zero-shot: the reference audio is a prompt, not training data. Practical tips from docs: 3 seconds works, but 5–15 seconds of single-speaker, close-mic, no music/noise/reverb yields better results. Output timbre and style match the reference. Built-in AudioSeal imperceptible watermark (enabled by default) marks synthetic speech for detection. Cloning requires explicit permission from the voice owner.

Video Dubbing: Multi-Language, Speaker Preservation

Workflow: upload video → transcribe → translate → preserve speakers → synthesize → export subtitled video. It detects number of speakers (diarization), assigns voices per speaker, translates to target language, and can retain background music. Supports 646 languages. README includes demos: English video dubbed to Spanish, French, Japanese, Chinese, etc.

Not Just a Desktop App: API / MCP Platform

VoiceStudio also serves as a local voice service:

OpenAI-compatible API: Change base_url from https://api.openai.com/v1 to http://localhost:3900/v1 to route existing OpenAI speech code to local engines for TTS and transcription.

MCP Server: Add an MCP endpoint so Claude Desktop, Cursor, etc., can call generate_speech, clone_voice, transcribe.

Agent skills: Install skills to let coding agents (Claude Code, Codex, Cursor) use VoiceStudio for synthesis and transcription.

Rust control sidecar: Enables Herdr, coding agents, VS Code, desktop apps to trigger system-wide dictation.

This makes VoiceStudio a "voice organ" for AI agents, not just a human-facing tool.

Technical Architecture: Building a Local Voice Platform

Architecture follows a "desktop shell + local service" layering with clear source structure:

Desktop shell (frontend/src-tauri/): Tauri v2 (Rust) manages window, tray, hotkeys, updater.

Frontend (frontend/src/): React + Vite UI, Zustand for state.

Backend (backend/): FastAPI service on localhost:3900, includes engine registry, dubbing/audio/long-audio pipelines, OpenAI-compatible API, MCP server.

Storage (omnivoice_data/): Projects, voices, settings, logs, SQLite state all on local disk.

Worker system: Authenticated remote compute for offloading heavy tasks.

This "local service" design lets VoiceStudio be both a desktop app and a callable local voice backend, enabling easy API and MCP exposure.

Dictation, Vocal Isolation, Speaker Diarization: Practical Details

Dictation component: System hotkey triggers real-time transcription; optional local LLM for text cleanup. Bound in Rust sidecar, invokable by coding agents and VS Code.

Vocal isolation: Uses Demucs to separate vocals from background music, useful for karaoke, sampling, cleaning assets.

Speaker diarization: Uses Pyannote/WhisperX to identify "who spoke when", valuable for meeting transcription, multi-speaker interviews, and dubbing speaker preservation.

These capabilities, each a standalone tool elsewhere, are integrated into one application.

MCP & Agent Skills: Giving AI Your Voice

VoiceStudio provides complete developer/agent integration. Two concrete examples:

MCP Server (for Claude Desktop, Cursor):

{
  "mcpServers": {
    "voicestudio": {
      "url": "http://localhost:3900/mcp"
    }
  }
}

After configuration, AI can call generate_speech, clone_voice, transcribe.

Agent skills (for Claude Code, Codex, Cursor): npx skills add debpalash/VoiceStudio Once installed, coding agents can synthesize speech and transcribe audio via local VoiceStudio.

Getting Started

Install via GitHub releases (macOS Apple Silicon DMG, Windows MSI, Linux AppImage) or Docker:

docker run -d -p 127.0.0.1:3900:3900 \
  -v omnivoice-data:/app/omnivoice_data \
  --name voicestudio palashdeb/omnivoice-studio:stable

First launch auto-creates Python environment and downloads default models. First voice clone: open Voice Cloning → add clean voice sample → enter text → generate. A Google Colab notebook is available for cloud trial without installation.

Limitations & Boundaries

Hardware requirements: Local large models need 16GB+ RAM, NVIDIA GPU (8GB+ VRAM) or Apple Silicon. CPU/low-spec can run with lightweight engines at quality cost. Intel Macs cannot run local backend (remote only).

Active beta: README advises using latest release for stable work; main branch may have breaking changes.

Model licenses: App is AGPL-3.0, but downloaded models retain upstream licenses. Default OmniVoice weights are CC-BY-NC (non-commercial); commercial use requires checking chosen model's terms.

Chinese docs: Full simplified Chinese README (README_CN.md) for Chinese users.

Positioning vs. ElevenLabs

VoiceStudio doesn't aim to replace ElevenLabs' cloud experience; it trades cloud convenience for local control:

Data path: VoiceStudio local by default; cloud services process audio/text on provider servers.

Cost: VoiceStudio free software, you provide hardware; cloud services subscription/credits/per-character fees.

Offline use: VoiceStudio works offline after model download; cloud services typically require internet.

Customizability: VoiceStudio open source, engines, models; cloud services limited to provider options.

Maintenance: VoiceStudio self-managed updates, disk, compute; cloud provider manages infrastructure.

In short: choose cloud for zero-setup convenience; choose VoiceStudio for data sovereignty, offline capability, deep customization, and high-frequency cost-free usage.

Who Should Use It

Privacy-conscious users: Don't want voice, recordings, text uploaded to cloud.

Content creators / high-volume dubbing needs: Frequent cloning, dubbing, audiobooks without per-character costs.

Developers: Want to add local TTS/STT to apps or agents; OpenAI-compatible API, MCP, skills ready.

Chinese users: Full Chinese docs, good Chinese voice support (OmniVoice 600+ languages, CosyVoice).

Design Philosophy Worth Remembering

VoiceStudio turns "local-first" from a slogan into engineering practice: data path defaults to local, telemetry off by default, all remote capabilities explicit opt-in. In an era where AI services "go cloud to harvest data," it proves the opposite direction is viable—your most sensitive personal data (your voice) can stay fully under your control.

It also positions smartly: not just a human-facing desktop tool, but simultaneously an API, MCP server, and agent skill, letting voice capabilities embed seamlessly into the AI ecosystem. As "AI's voice" becomes infrastructure, VoiceStudio has already built that infrastructure on your own machine.

This is open source's greatest value: it returns choice to users—you can pick cloud for convenience, or local for freedom, and VoiceStudio makes the latter truly complete and usable.