How Ant Group Supercharged Front‑End AI with Cross‑Platform Smart Apps

Qingbi, a frontend engineer from Ant Group, presented at SEE Conf 2022 about the application of cross‑platform intelligent services.

What is a smart application?

A smart application combines a regular app with an intelligent service; the service can run in the cloud (cloud AI) or on the device (edge AI). Edge AI offers high real‑time response, privacy protection, and reduced cloud compute costs.

High real‑time – data generated on the device can be processed instantly without sending it to the cloud.

Privacy protection – all data is consumed and discarded on the device, preventing exposure.

Save cloud resources – computation is distributed to many devices, lowering overall cost.

What is cross‑platform intelligence?

Cross‑platform intelligence means a single JavaScript codebase can be deployed to any JS environment (Alipay, Taobao, WeChat, etc.) without relying on native client capabilities.

Business case 1: Pet Camera

Pet nose prints are unique, enabling pet identity records for insurance and loss‑prevention. Manual photo uploads had low success rates. By deploying a cross‑platform smart service inside the mini‑program, the camera captures a suitable pet photo in real time, raising the success rate to over 80% across Alipay, Taobao, Xianyu, and WeChat.

Business case 2: Screen‑break Insurance Camera

When users manually upload screen photos, poor quality images hinder AI assessment of broken screens. The smart camera guides users to capture clear, properly sized screen images, which are then evaluated in the cloud, enabling reliable insurance claims.

Technical challenges

Front‑end high‑performance computing is constrained by limited device hardware, especially on low‑end Android phones. Real‑time inference requires fast processing; otherwise user experience suffers.

How we solved them

The smart service consists of an AI model and a compute engine. The initial engine used TensorFlow.js, supporting only about 30% of devices.

We switched to a WebGL‑based engine to leverage GPU acceleration. By reorganizing matrix storage into block layouts, we reduced cache misses, and we further optimized the model graph, applied vectorization, and used mixed‑precision calculations.

These optimizations doubled engine performance and increased device coverage from 30% to 93%, enabling successful product rollout.

Future outlook

Ant Group is building its own cross‑platform engine, improving size and performance, and plans to open‑source the engine and related ecosystem to the community.