How Alibaba’s AliPlayStudio Powers Real‑Time AI Video Interactions on Mobile

Background

Alibaba Search and Zhejiang University jointly built the AliPlayStudio video‑interactive platform, deploying online (Taobao app) and offline (large‑screen) scenarios. By integrating brand resources and AI‑driven video interactions, the platform converts engaged users into consumers through games, face‑fusion, and personalized recommendations.

Human Semantic Segmentation

Human parsing separates body parts at pixel level. Data is augmented via image synthesis with color transfer and realistic human placement, increasing sample volume dramatically. The high‑precision model uses an Inception backbone with ASPP, while the real‑time model employs a lightweight encoder, aggressive channel pruning, fast down‑sampling, and variable input sizes. Decoder follows a UNet‑style design with feature fusion and residual connections. The final model (~1.7 MB, 0.5 MB after quantization) achieves 0.94 mIoU at 25 FPS on a Qualcomm 625 device (320×240 input).

Gesture Recognition for Rock‑Paper‑Scissors

A real‑time hand‑gesture game was launched during Double‑11 2018. Using an SSD‑based detector with an MNasNet‑derived backbone, feature‑pyramid fusion, and knowledge‑distillation, the model size is 1.9 MB, runs in 17 ms on iOS devices, and reaches 0.984 AP (IoU=0.5).

Human Pose Estimation

The pose model adopts an encoder‑decoder architecture with MobileNet as backbone and a PAF‑based decoder inspired by OpenPose. It runs at 11 ms per 320×320 frame on Snapdragon 845 and 15 FPS on RK3399, with a 2.5 MB footprint.

Controllable Image Style Transfer

A stroke‑pyramid network enables continuous brush‑size control. Multiple branches with varying receptive fields are trained on different style scales; during inference a gating function selects the appropriate branch. The model (0.99 MB) processes a 1024×1024 image in 0.09 s on an NVIDIA Quadro M6000.

Face Fusion

The system aligns facial landmarks, applies pose‑aware warping, and normalizes skin tone using a predefined LUT to handle diverse lighting conditions. This yields high‑quality face‑fusion results even for low‑quality selfies.

Conclusion

Since March 2018, AliPlayStudio has powered over ten AI‑interactive marketing campaigns across mobile and offline channels, combining cutting‑edge computer‑vision techniques with efficient model design to deliver engaging experiences on low‑end devices.

References

Howard et al., MobileNets, 2017.

Chen et al., DeepLab, 2018.

Chang & Chen, Pyramid Stereo Matching Network, 2018.

Li et al., Pyramid Attention Network, 2018.

Gong et al., Instance‑level Human Parsing, 2018.

Liu et al., SSD, 2016.

Sandler et al., MobileNetV2, 2018.

Tan et al., MnasNet, 2018.

Lin et al., Focal Loss, 2018.

Lin et al., Feature Pyramid Networks, 2017.

Ren et al., Faster R‑CNN, 2015.

Cao et al., Real‑time Multi‑person Pose, 2017.