Accelerating Frontend AI: From WebGL to MNN.js and Beyond

Frontend AI During the Pandemic

AI has been booming in the front‑end for a few years. The pandemic brought AI to the forefront, exemplified by a WebApp that adds masks to faces using faceapi and a mask library, which went viral on Zhihu and social circles.

Why Front‑End AI Lags Behind Native

Since 2017, many mobile deep‑learning frameworks have emerged, but the front‑end mainly relies on TensorFlow.js. ONNX.js remains immature, and WebNN is still under standardization, limiting its excitement compared to native side AI.

WebNN and Native Acceleration

WebNN prefers platform‑provided accelerators such as DirectML or NNAPI, falling back to WebGL, WebGPU or WebAssembly when necessary. If adopted as a browser standard, it could enable low‑cost compute acceleration, though broad platform support remains a challenge.

Case Study: MNN.js

In a Taobao AR mini‑program, TensorFlow.js struggled on mid‑range devices (e.g., Mate 30, iPhone 7). Bridging to the native MNN engine achieved a 4× frame‑rate increase and significant memory savings across all devices.

Mini‑program frame rate caps at 30 fps; native MNN exceeds this limit.

Front‑End Acceleration Toolbox

JavaScript (asm.js) – 100 % coverage, but performance is dozens of times slower than native.

WebGL / WebGL2 – Browser coverage ~97 % / 74 %; good performance for tfjs and onnxjs, yet many low‑end Android devices lack required extensions.

WebAssembly – ~91 % coverage in browsers and full support in major mini‑programs (WeChat, Alipay, Taobao); performance still lags native due to missing multithreading, SIMD, and 64‑bit support.

WebGPU – Promising but currently limited coverage; performance alone cannot compensate for its scarcity.

Combining WebGL and WebAssembly provides broad coverage and respectable performance for most browsers and mainstream mini‑programs.

Building a Faster Blade: MNN.js

By reusing MNN’s model format, MNN.js supports conversion from Caffe, TensorFlow, and PyTorch models, and inherits operator replacement and layer‑fusion optimizations. It offers three back‑ends (WebGL, WebAssembly, JavaScript) with WebAssembly as a fallback when WebGL is unavailable.

MNN.js currently supports 38 operators and delivers over 35 % inference speed‑up compared to TensorFlow.js on a variety of devices, while maintaining compatibility across Chrome, Safari, and major mini‑program environments.

async function loadModel() {
  let compiled = await MNN.WasmBuilder.compile(fetch('/path/to/mnn.wasm'));
  let wasm = await MNN.WasmBuilder.create(compiled);
  let js = MNN.JSBuilder.create();

  let builders = [webgl, wasm, js];
  let result = await fetch(url);
  let model = await result.arrayBuffer();
  return MNN.loader.loadMNN(new Uint8Array(model), builders);
}

async function run(webcam) {
  let width = webcam.videoWidth;
  let height = webcam.videoHeight;
  let preprocess = {
    sourceFormat: 'rgba',
    destFormat: 'bgr',
    filterType: 'nearest',
    wrapType: 'zero',
    cropAndResize: [[0, 0, width, height], [0, 0, 224, 224]],
    means: [103.94, 116.78, 123.68],
    norms: [1 / 127.5]
  };
  let rgba = capture(webcam);
  await model['data'].setData(rgba, "uint8", [1, height, width, 4], "NHWC", preprocess);
  let result = await model['prob'].getData("NHWC");
  let values = new Float32Array(result.data);
  let sorted = values.slice().sort((a, b) => b - a);
}

Where Is Front‑End AI Heading?

Deep learning’s three pillars—data, algorithms, and compute—continue to evolve, but successful AI products still require business‑driven scenarios. Mini‑programs now break the app‑island barrier, allowing AI‑enhanced shopping experiences such as face, pose, and gesture recognition that engage users while they browse.