How Edge AI Agents Run on Phones and PCs: Architecture Deep Dive

Large language models are powerful, but for agents to become everyday tools they must run on edge devices such as phones, PCs, cars, and IoT gadgets. Edge AI faces three core challenges: limited compute, power‑sensitive operation, and complex deployment. This article dissects the three foundational layers that address these challenges.

First Layer – Unlocking Compute

The Arm SME2 instruction set is integrated into the MNN inference engine, providing end‑to‑end image encoding and decoding acceleration. By leveraging SME2, the same model can execute faster and with lower energy consumption on Arm‑based edge hardware.

Second Layer – Slimming the Model

Tencent’s mixed‑precision quantization technology (2‑bit / 1.25‑bit) dramatically reduces model size, allowing large models to fit on resource‑constrained devices. The article also explains how the Qwen‑Omni model is re‑architected for multimodal interaction on the edge.

Third Layer – Deploying the Solution

ExecuTorch is used for practical edge AI deployment on Arm platforms, demonstrating real‑world integration steps. Additionally, the emergence of Windows on Arm creates new opportunities for AI‑enabled PCs, expanding the deployment landscape for edge agents.

The content originates from the Edge AI track of Arm Create 2026 (Shanghai on September 9 and Shenzhen on September 11), where engineers from Arm, Tencent, Alibaba, and other frontline teams delivered over 30 hands‑on sessions.