Exploring AR Integration in Baidu Mini Games: Techniques, APIs, and a Development Case Study

This article documents a talk from the GMTC Mobile AI session titled “Exploring AR in Baidu Mini Games”. It is divided into three parts: an overview of typical AR applications, the browser‑based AR integration approach, and a concrete case study of building an AR demo inside a Baidu mini‑game.

1. Typical AR Application Examples

The speaker presents six representative AR cases:

Marker‑based tracking (e.g., AR.js using ARToolKit) where a barcode‑like image is used as a marker to generate a pose matrix for 3D overlay.

Image‑based tracking that works with natural images (e.g., a Mario game box cover) to attach virtual content.

Plane tracking (ARKit / ARCore) demonstrated by an Angry Birds AR version that detects horizontal/vertical planes.

Location‑based AR games such as Pokémon Go, which rely on LBS.

Snapchat‑style AR marketing that recognizes landmarks, performs sky segmentation, and plays a sky‑box video.

Face‑enhancement AR that detects facial landmarks, builds a 3D face mesh, and applies filters or stickers.

2. Browser‑Based AR

In browsers, AR capabilities are exposed through events like deviceorientationevent and devicemotionevent for sensor data, and WebRTC for camera streams. Deep‑learning libraries such as TensorFlow.js and Baidu’s Paddle.js can run inference on the camera feed. However, JavaScript performance on mobile is limited: no SIMD, limited multithreading, and reliance on WebGL for GPU acceleration. Emerging standards such as WebGPU , WebNN , and the WebXR immersive group aim to provide more efficient low‑level access. Baidu mini‑games adopt the WebXR spec for future AR compatibility.

3. AR Integration in Baidu Mini Games

The mini‑game runtime includes a V8 (Android) or JavaScriptCore (iOS) engine, WebGL/Canvas rendering, and system APIs for file, network, etc. The AR stack combines three sources:

ARKit / ARCore capabilities : World Tracking – provides a real‑time view matrix. Point Cloud – exposes per‑frame feature‑point coordinates. Plane Tracking – detects horizontal and vertical planes. Hit Test – casts a ray from a screen point to intersect detected geometry. Anchor – a fixed pose in world space that can be attached to scene nodes. Light Estimate – returns ambient intensity and RGB color for realistic lighting.

DuMix AR (Baidu’s own SDK) for face‑related enhancements: Camera data exposed as a Video object usable as a WebGL texture. 95 facial landmark points returned in a Float32Array . Face skeleton matrices for driving a 3D face mesh. Expression coefficients (0‑1 values) for mouth, eyes, eyebrows, etc.

4. Development Case: An AR Car Demo

The demo uses THREE.js to load a glTF car model, then switches to AR mode via the mini‑game API:

swan.requestXRSession({
  // optional parameters such as requiredFeatures
});

After the session is created, each frame is obtained:

const frame = session.getFrame();
const planes = frame.getTrackablePlanes();

When a suitable plane is found, a hit test is performed and an anchor is created to place the car model:

const hit = frame.hitTest(x, y);
if (hit) {
  const anchor = hit.createAnchor();
  // attach model to anchor.poseMatrix each frame
}

Rendering optimizations include:

HDR environment maps with tone‑mapping using the light‑estimate values.

Physically‑based rendering (PBR) for metallic/roughness and clear‑coat effects.

Shadow techniques: static contact shadows, screen‑space ambient occlusion (SSAO), and a cheap Gaussian‑blurred plane shadow for the car’s underside.

These steps produce a convincing AR experience where the virtual car appears anchored to a real table, with lighting and shadows matching the surrounding scene.

Summary

The talk covered AR application patterns, the current state of web‑based AR standards, and the concrete integration of ARKit, ARCore, and Baidu’s DuMix SDK into the Baidu mini‑game platform. By unifying concepts such as world tracking, plane detection, anchors, and light estimation, developers get a consistent front‑end‑friendly API. Future work will add body‑tracking, hand‑gesture, and more deep‑learning‑driven perception to further enrich AR experiences on mobile.