How Alibaba’s Mini‑Program Powers AR Makeup Try‑On with MNN & TensorFlow.js

What is AR?

AR (Augmented Reality) overlays virtual effects onto the real world. By adding AR capabilities to merchant apps, brands can enrich the user experience.

Collaboration with Modiface

Before Double 11, Alibaba’s merchant app team partnered with L’Oréal’s Modiface to launch AR makeup try‑on engines for YSL and Armani flagship stores on the Taobao mini‑program platform.

Link Overview

Brands provide design and interaction, Modiface supplies the AR engine built on the merchant app’s basic capabilities, and the combined solution is deployed to the brand’s store.

Technical Architecture

Standardized API and component capabilities are offered. The AR engine runs mainly in JavaScript, using MNN or TensorFlow.js for inference, WebGL for rendering, and native camera components for frame capture.

AR Try‑On Process

Load Modiface plugin, open camera, receive native frames, initialize WebGL canvas.

Run contour detection model (TensorFlow.js) and landmark detection model (MNN).

Switch to landmark detection after face detected.

Capture ambient light to adjust beautification.

Render makeup on facial regions via WebGL shaders.

If no face detected, skip rendering.

Core Capabilities

Three main capabilities: real‑time camera frame, deep‑learning inference acceleration, and rendering.

Real‑time Camera Frame

Provides ArrayBuffer frame data via a native Camera component embedded in WebView, using an off‑screen FBO for YUV/RGB conversion.

Supports up to 30 FPS on low‑end devices.

Deep‑Learning Inference Acceleration

Supports MNN and TensorFlow.js; MNN offers better speed and lower memory usage.

Rendering Capability

Canvas component backed by GCanvas provides WebGL 1.0 and Canvas2D APIs.

Capability Evolution

Future plans include migrating the engine to WebAssembly, adding WebGL 2.0 and Metal/Vulkan support, sharing camera textures to reduce JS‑Native communication, and expanding features such as target tracking, image recognition, gesture detection, pose detection, space recognition, and face detection.

Experience

Users can try the YSL and Armani AR makeup apps via the respective flagship store pages.