Voice Conversion: Fundamentals, Methods, and iQIYI Applications

Voice Conversion (VC) transforms a speaker's voice to another timbre while preserving the linguistic content, enabling applications such as dubbing, entertainment, and medical assistance. Early VC relied on parallel corpora to learn mappings between source and target speakers, but parallel data are hard to obtain.

Recent methods use non‑parallel data, often leveraging ASR‑derived features like Phonetic Posteriorgrams (PPG) or bottleneck features, allowing many‑to‑one or many‑to‑many conversions. Classic frame‑wise conversion uses Dynamic Time Warping (DTW) for alignment and Gaussian Mixture Models (GMM) for mapping, as implemented in the open‑source tool Sprocket.

Sequence‑to‑sequence models with attention address variable length alignment, while Maximum Likelihood Parameter Generation (MLPG) improves quality by jointly modeling static and dynamic features. Variational Autoencoders (VAE) encode speech into latent variables, discarding speaker identity, and decode with target speaker conditioning, often using a cyclic reconstruction to reduce information loss.

Normalizing‑flow based models such as Flow‑based VAE (e.g., Blow) apply invertible transformations to map source speech to a latent space and back, enabling flexible many‑to‑many conversion. Generative Adversarial Networks (GAN) like CycleGAN and StarGAN are also employed; CycleGAN uses two generators and discriminators to enforce cycle consistency, while StarGAN adds conditional inputs for many‑to‑many mapping.

iQIYI adopts many‑to‑one or many‑to‑many strategies to reduce user constraints, enhancing prosody encoding, pitch contour modeling, and data augmentation to improve robustness in diverse scenarios such as narration and singing. The company also explores VAE, Flow, and GAN variants for higher quality conversion.

The article concludes with a Q&A covering detection of synthetic speech (referencing the ASVspoof competition), the limited benefit of VC for augmenting ASR training data, cross‑language VC preserving rhythm, and the current state of TTS technologies.