Technical Overview of Youku's Video Face Swapping System

Youku recently launched a video face‑swapping activity that allows users to upload a single photo and replace a celebrity's face in short video clips with their own. The service aims to provide a low‑threshold, entertaining experience where users can see themselves acting in scenes alongside or instead of their favorite stars.

The article reviews the evolution of face‑swapping technologies, from early manual Photoshop techniques to modern deep‑learning based Deepfakes, and explains why these methods are considered near‑photorealistic.

Key technical goals of Youku's system are:

1. Single‑image input – only one user photo is required. 2. Fast generation – the same model is used for all users without per‑user retraining. 3. General‑purpose content – the system can handle various angles, expressions, and lighting conditions.

To achieve these goals, the team employed a 3D generative model combined with several auxiliary techniques, addressing challenges such as:

• Large variations in human appearance (age, skin tone, facial features) that make a single model insufficient. • Accurate facial region segmentation in videos, especially under occlusions (hands, props, hair, glasses, etc.). • Missing facial details in the source photo (closed mouth, profile view) that must be inferred. • Temporal consistency across frames, preventing jitter caused by frame‑wise random generation. • High‑resolution output, targeting at least 256×256 pixels to match 540p video quality.

The solution includes:

• Defining multiple face categories (gender, age range, appearance type) and training separate models for each. • Using deep‑learning based segmentation with skin color and facial landmarks to create precise masks. • Applying a 3D face model to estimate the height of facial features and project them back to 2D video frames. • Incorporating open‑mouth training data to generate realistic teeth when needed. • Employing feature‑point matching and filtering to smooth trajectories and maintain temporal stability. • Using a multi‑scale + super‑resolution pipeline: first generate 128×128 faces, then upscale to 256×256 with a super‑resolution network.

The overall pipeline consists of per‑frame face detection, segmentation, landmark extraction, trajectory smoothing, 3D modeling, generation, super‑resolution, and finally image fusion to produce the swapped video.

Current limitations include lack of beauty filters, inaccuracies with large side‑view angles or heavy occlusions, and the fact that a single generic model is used for all users, leading to less personalized results compared to per‑user trained Deepfakes.

Despite these issues, the technology demonstrates significant potential for both user‑generated content and future applications such as generating synthetic performances by unavailable or uncooperative celebrities.