AI‑Powered Photo‑to‑3D Avatar Generation in Taobao Life 2

Taobao Life 2 (also known as Second Life) is a popular dress‑up application that offers a core "photo‑face" feature: users can upload a portrait and instantly obtain a personalized 3D digital avatar.

The article introduces the AI‑driven workflow behind this feature, describing the technical challenges, the overall framework, and the evaluation methodology.

Background : Manual face‑sculpting in the app involves more than 80 sliders and can take up to half an hour. To improve efficiency, the team explored a one‑click solution that generates a 3D avatar from a single 2D photo.

Key Challenges (1) mapping 2D facial features to a 3D model, (2) preserving the stylized cartoon base while retaining user‑specific details, and (3) incorporating discrete attributes such as glasses, hairstyle, and beard.

Technical Framework : The pipeline consists of four modules – preprocessing, facial attribute extraction, non‑facial attribute extraction, and avatar generation. The preprocessing stage validates image quality and masks sensitive content. Facial attributes are recovered using a 3D Morphable Model (3DMM) that solves for shape and texture coefficients by iteratively fitting the projected 3D face to the 2D image. To accelerate the process, a lightweight MLP maps the 3DMM coefficients to the stylized avatar coefficients.

Non‑facial attributes (glasses, hair, eyebrows, beard) are classified at a fine‑grained level and matched to corresponding asset IDs, enhancing the realism of the final avatar.

Evaluation : The team adopts the Normalized Mean Error (NME) metric, a standard measure in 3D face reconstruction, combined with subjective scoring. Reported NME values for several state‑of‑the‑art methods (e.g., 3DDFA + SDM: 3.43, BCLL: 2.47) serve as baselines. The Taobao Life 2 solution achieves a significant reduction in generation time—from ~30 minutes manually to <5 seconds automatically—while delivering avatars that are both “like” and “beautiful”.

Results & Future Work : The current system supports only the female cartoon base; a male version is under development. Further improvements target finer facial detail fitting and expanded modality support (text‑ or voice‑driven avatar creation).