How Adaptive 3D Face Cutout Transforms Kuaishou’s AR Effects

Abstract

Adaptive 3D face cutout is the core technology of the new 3D zoom face effect template on the Beyond Effects platform. During the Spring Festival and Winter Olympics, effect creators used this template to produce many high‑quality video effects. This article introduces the problems solved by the adaptive 3D cutout technique and its technical details.

Background

Beyond Effects (formerly Kwai Studio) serves Kuaishou’s shooting products, providing both professional and commercial creative support for C‑ and B‑end users. It offers beauty, filter, material configuration, and various visual‑effects tools that creators can use to build their own effects.

Kuaishou’s video‑effect business integrates 2D/3D effects, beauty, and image AI technologies across many apps.

The Y‑tech team leverages graphics, image‑processing, and deep‑learning techniques—especially face and body detection, key‑point extraction, and segmentation—to create immersive effects such as stickers, frames, accessories, makeup filters, and motion games.

3D Zoom Face Effect

The “3D Zoom Face” effect applies adaptive 3D cutout technology, combining 3D face mapping and custom camera control. It allows the rendered face to change position, size, angle, and quantity, and modifies scene perspective and camera parameters to achieve dynamic camera movements.

This capability expands creative space for designers, lowers shooting cost and threshold for users, and enables immersive experiences such as multi‑person role‑play, gender‑based interactions, and motion‑triggered games.

Implementation Details

Effect creators can download the Beyond Effects tool from https://effect.kuaishou.com and start from the “3D Zoom Face” template (e.g., the “Iron Pot Stewed Goose” effect) or create a new project and add the template from the 3D category.

Users can experience the effect directly in Kuaishou or Kuaishou Lite apps. Sample works demonstrate immersive scene replacement, customizable camera angles for advertising, complex facial animations, and gender‑aware multi‑character play.

Technical Framework

The adaptive 3D cutout pipeline consists of three steps:

AI face detection on the phone’s input frame to obtain facial landmarks.

Render the face onto a standard 3D model using a sampling camera and store it as a RenderTarget.

UV‑map the sampled face onto the final mesh and render it with custom face‑mapping and camera‑control scripts.

When the face moves toward the screen edge, the sampling camera may capture only a partial region, causing stretching artifacts. Two optimization schemes are proposed: moving the camera or rotating it to keep the face within the rendering window.

Rotating the camera avoids sampling errors but can introduce perspective distortion; moving the camera avoids distortion but may cause sampling gaps. In practice, the platform prefers the rotation scheme with moderate FOV settings.

Custom Camera Control Solution

The solution computes the required camera rotation angle based on phone FOV, custom camera FOV, distances, and the face’s position relative to the screen edges. The Y‑tech AI model detects 101 facial key points; a subset of seven points (ears, chin, nose tip, forehead) is used to estimate face position and fill‑level.

When the face is near the screen border, the camera is rotated or moved to keep the face fully within view. Formulas (omitted for brevity) calculate the maximum correction angle and weight based on face fill‑level.

Conclusion and Outlook

The “3D Zoom Face” effect greatly expands creative possibilities while reducing user effort. Future work will explore more high‑quality, fun templates and continue to support effect creators on the Beyond Effects platform.