How JD Mini‑Program Achieved Smooth AR Face‑Tracking on Low‑End Phones

Background

JD Mini‑Program is an open technology platform increasingly adopted by top brands for private‑domain traffic marketing and operations. Brands in cosmetics, luxury goods, etc., demand AR/VR capabilities like virtual try‑on, prompting the JD engine to provide low‑level APIs for customized development.

Technical Solution

We collaborated with the AR/VR team to design and develop the required capabilities, which were released in JD APP version 11.6.6.

Concept Introduction

Key technologies: camera, real‑time frames, AR algorithms, same‑layer rendering, WebGL. The camera captures real‑time face frames, AR algorithms compute results, and data is passed to the front‑end. WebGL, implemented via native OpenGL, enables high‑performance rendering for games and AR/VR. Same‑layer rendering mixes native components with the WebView DOM to keep rendering order, scrolling, and touch handling consistent.

Overall Flow

The engine exposes JavaScript APIs for camera access, frame acquisition, AR processing, and WebGL rendering. Developers invoke these APIs to start the camera, obtain frames, call AR interfaces, and render results.

Layered Design

The architecture separates the AR engine layer into built‑in and external engines, allowing third‑party AR solutions while providing atomic capabilities such as camera, real‑time frames, and WebGL.

Technical Challenges

Challenge 1: Massive data transfer between native and JavaScript becomes a bottleneck.

Challenge 2: iOS camera output is BGRA, while WebGL expects RGBA, requiring costly format conversion.

Challenge 3: Each frame must be processed within ~41 ms, demanding careful concurrency and ordering to avoid visual glitches.

Performance Optimizations

1. Data Transmission Optimization

We introduced a NativeBuffer cache to keep a single copy of camera frames. Using JavaScriptCore’s “NO COPY” API, a native pointer is wrapped as an ArrayBuffer or TypedArray, sharing memory between native and JS.

When the JS object is destroyed, a C deallocator callback frees the native buffer.

typedef void (*JSTypedArrayBytesDeallocator)(void* bytes, void* deallocatorContext);

2. Write Optimization

Data flows bidirectionally. To avoid unnecessary copies when JS sends data back to native, we retrieve the raw memory address of the JS ArrayBuffer and reuse the existing NativeBuffer if possible.

3. Data Type Consistency

We enforce a single binary type (e.g., ArrayBuffer) throughout the pipeline; converting to TypedArray would break the write‑optimization and cause stutter on low‑end devices.

4. Camera Frame Format Conversion

iOS provides BGRA frames, but WebGL requires RGBA. By swapping the first and third bytes of each 4‑byte pixel (without extra allocation), we efficiently convert BGRA to RGBA.

5. Concurrency Scheduling

We offload camera capture, format conversion, AR processing, and WebGL rendering to separate serial queues, keeping the UI thread free. Thread‑safe guards protect the NativeBuffer pool.

6. Resource Management

We limit the buffer pool size and mark buffers as “consumed” to avoid premature deallocation. On memory warnings or when the pool is full, unused buffers are cleared. The engine also stops frame listeners when the page exits to prevent duplicate processing.

Conclusion

The JD Mini‑Program platform continues to evolve its open capabilities, focusing on performance and user experience. We invite collaboration and feedback from other teams to further enhance the ecosystem.