How Alibaba’s MediaAI Studio Brings AI‑Powered Live Stream Interactions to Life

Introduction

Pan Jia (aka Lin Wan) from Alibaba Taobao multimedia front‑end presents "Media Intelligence – Taobao Live Stream Interactive Media" in five parts: how anchors greet users in a live room, how to create a gesture‑greeting effect, the overall MediaAI solution design, the MediaAI Studio editor implementation, and future construction directions.

How to greet in the live room?

During the Chinese New Year, anchors can perform a greeting gesture that triggers festive visual effects such as animated text, couplets, fireworks, or face accessories like a "wealth‑god" hat. The system recognizes the gesture and the anchor’s face in real time to render these effects.

Creating the gesture‑greeting effect

The production workflow consists of four steps: (1) designers create static or animated assets (e.g., a wealth‑god hat) using design software; (2) assets are assembled into a package in the self‑developed MediaAI editor, where frame adaptation, face‑following, gesture trigger conditions, and local preview are configured; (3) the package is uploaded to the content platform; (4) anchors select and enable the asset package during streaming, where real‑time recognition and rendering occur.

Example: adding a sticker, uploading a sequence‑frame image, adjusting position and size, setting the trigger to the greeting gesture, and previewing the effect on a prepared video.

Another example shows adding a wealth‑god hat sticker that follows the anchor’s forehead when the anchor nods.

Media intelligence solution design

Traditional "red‑packet rain" interactions overlay an H5 page on the video stream, which is disconnected from the stream content. Media intelligence embeds interactive assets directly into the video stream, allowing anchors to control the rain of red packets via gestures, thereby increasing interaction rate and viewer dwell time.

The solution consists of two parts: intelligent assets and interactive gameplay. Intelligent assets provide a one‑stop platform for designers to produce face filters, stickers, and other effects using a JSON‑based module configuration. Interactive gameplay offers developers a code‑centric IDE for creating stream‑level interactions.

The production‑to‑consumption pipeline includes four stages: asset production, asset management, asset usage, and asset display. Producers use the editor to create gameplay assets, which are managed on a material platform and integrated into the ALive component system. Anchors enable gameplay via a control console; the streaming side merges the assets into the video stream, transmitting position and hotspot data via SEI keyframes for client‑side interaction.

MediaAI Studio editor

MediaAI Studio is a desktop editor built on Electron. Its rendering engine, RACE, integrates the MNN inference framework and the PixelAI algorithm platform to provide face and gesture detection, rendering, and compositing capabilities.

The Electron main process handles window management and updates, while the renderer process hosts the module tree, editing panels, and real‑time preview. A worker thread communicates with the RACE native module via JSON and binary protocols, enabling high‑performance rendering.

Developers can write gameplay scripts in JavaScript that invoke the RACE C++ module through N‑API bindings, achieving near‑native performance for interactive games.

Future construction

Media intelligence is still in its early stage. Future work will deepen integration with algorithm, material, and publishing platforms, enforce front‑end security standards for JavaScript scripts, and provide ecosystem support for designers and ISVs. The goal is to accelerate the creation of new live‑stream interactive experiences and expand the variety of gameplay types.

Q&A Highlights

Front‑end responsibilities include building the MediaAI Studio editor (Electron), integrating the editor with ALive, providing streaming tools for anchors, and delivering interactive components in the live room.

Detection frequency is controlled by enabling/disabling gameplay packages and by per‑algorithm frame settings to balance performance.

Recognition and compositing run on the anchor’s streaming client (both PC and app) using standard RTMP/HLS/HTTP‑FLV protocols.

Compositing does not add latency; performance issues affect frame rate. Critical sync scenarios use SEI+CDN to keep video and data aligned.

Open‑source gesture detection library: Google MediaPipe (https://github.com/google/mediapipe).

Algorithm models are executed on the device via MNN and PixelAI, not bundled in the front‑end package.

RACE C++ code is exposed to JavaScript via N‑API, offering near‑native execution speed for game logic.

Red‑packet positions are random; hotspots are defined by SEI keyframes transmitted from the streaming client.

Game performance relies on RACE’s native rendering and upcoming WebGL interfaces to integrate with mainstream H5 game engines.