How ACE Powers Edge AI: A Heterogeneous Compute Engine for Real‑Time Inference

Background

In the AI field, insufficient chip compute prevents the deployment of autonomous driving and wearable devices. ACE (AI Labs Compute Engine) is an edge‑side heterogeneous compute engine that supports cloud‑edge model management and accelerates workloads on GPU, DSP, and VPU using Google UINT8 quantization and Facebook QNNPACK.

1.1 No Chip, No AI

AI development relies on chips for algorithm implementation, massive data handling, and compute power. The market distinguishes AI chips by training vs. inference and cloud vs. edge, forming four quadrants:

Cloud training – dominated by NVIDIA GPUs.

Cloud inference – dedicated chips such as Google TPU, Intel Nervana, Cambricon MLU100, and Ali‑NPU.

Edge inference – injecting AI compute into edge devices is a growing trend.

Edge training – federated learning and on‑device distributed training protect privacy and enable personalization.

1.2 Why Edge Computing?

Edge computing offers low latency, bandwidth savings, offline capability, and privacy protection, making it ideal for video surveillance, autonomous driving, and other latency‑sensitive tasks.

1.3 Why Build a Custom Algorithm Engine?

Edge devices (cameras, robots, wearables) have limited compute resources. A custom engine abstracts hardware details, optimizes limited resources, and accelerates business logic.

Architecture Overview

Compute Engine

Compute layer – model quantization, heterogeneous acceleration, memory‑friendly design, assembly optimizations.

Access layer – graph‑based orchestration, common operators, reduced development cycle.

Model Management

Cloud side – integrates with AutoAI to generate mobile models.

Edge side – receives cloud commands and pushes updates.

Compute Engine Details

3.1 Compute Layer

Model quantization reduces resource and memory usage compared to float32. Initial low‑end chip tests showed a float32 model taking several hundred milliseconds; after quantization and optimization, single‑core latency dropped to 59 ms (17 fps) and further to 41 ms (3.17× speed‑up, 74 % memory reduction). Standard MobileNet‑v2 quantized on a single core achieved 2.2× acceleration, and two‑core parallelism reached ~3×.

3.2 Heterogeneous Acceleration

Combining CPU with specialized accelerators (GPU, VPU) balances workload and reduces power consumption. Example: a pen‑tip detection algorithm ran in 260 ms on 4 CPU threads (CPU usage >240 %). Using CPU+GPU cut latency to 150 ms and CPU usage to 50 %. Using CPU+VPU further reduced latency to 51 ms while saving CPU cycles and power.

3.3 Access Layer

The access layer simplifies algorithm development and speeds up deployment through:

One‑stop AutoAI integration for model training, graph construction, and management.

High‑level and low‑level operator libraries co‑developed with algorithm teams.

API/UI for graph building, packaging models and configs into single files.

Support for mixed deep‑learning and traditional algorithm graphs, performance analysis, debugging, and evaluation.

Model Management

4.1 Cloud Model Management

The cloud backend controls edge models, offering query, download, reload, and reset operations.

4.2 Edge Model Management

Beyond single‑model handling, ACE introduces a business dimension: multiple services can share a model (e.g., pet detection and gesture recognition), enabling many‑to‑many relationships between models and business logic.

Future Outlook

ACE aims to bring fast, accurate AI to devices like Tmall Genie and robots, improving usability, optimizing low‑level performance, and deepening hardware‑software collaboration to make the most of limited edge compute resources.

References

[1] https://arxiv.org/abs/1712.05877 [2] http://speak.clsp.jhu.edu/uploads/publications/papers/1048_pdf.pdf [3] https://code.fb.com/ml-applications/qnnpack/ [4] https://arxiv.org/pdf/1902.01046.pdf [5] https://arxiv.org/abs/1603.05279 [6] https://cloud.tsinghua.edu.cn/f/a0785cec353a4cd18d7d/ [7] https://www.leiphone.com/news/201809/ICs9ETzP7gPDEAkJ.html