Tencent AI Lab's Advances in High‑Fidelity 3D Face Digitization and Evaluation

The presentation introduces Tencent AI Lab's latest work on high‑efficiency 3D facial digitization, covering methods that use a single photo, multiple photos, and RGB‑D selfie capture to reconstruct realistic 3D face models.

It outlines a complete production pipeline: light‑field scanning, high‑poly reconstruction, defect cleaning, UV mapping, low‑poly retopology, texture and normal map generation, followed by dynamic modeling steps such as expression capture, controller binding, and final rendering.

To improve production speed, three solution tiers (A, B, C) are explored. Tier A replaces costly 360° light‑field rigs with a simple camera array and AI‑driven generation; Tier B targets consumer‑grade devices; Tier C achieves reconstruction from one or a few photos, making the technology accessible to end users.

The RGB‑D selfie workflow is detailed in seven steps: automatic frame selection, initial model fitting, differentiable‑rendering optimization, texture/normal map synthesis, head/attachment mounting, AutoRigging, and text/voice‑driven animation. Each step is illustrated with accompanying images.

For evaluation, the authors identify shortcomings in traditional rigid‑alignment and nearest‑neighbor error metrics, then propose improvements: local mask‑based alignment and a reverse non‑rigid deformation to refine correspondence. They introduce the REALY benchmark, containing 100 image‑model pairs with per‑region masks, enabling fine‑grained assessment.

Extensive experiments on open‑source single‑image 3D face reconstruction methods demonstrate the effectiveness of the new metrics. The work has been published at ECCV and TOG, with code and data publicly released.

The session concludes with a Q&A covering lighting correction, hair modeling, oral cavity handling, and factors that make their digital humans more realistic than competing approaches.