Heterogeneous Acceleration for Deep Learning: From CPU Limitations to AI Processors

Kevin Xiaoyu, a senior researcher at Tencent TEG Architecture Platform Department, focuses on heterogeneous computing for deep learning, FPGA cloud, and high‑speed visual perception. This article is the first of a three‑part series analyzing the evolution of heterogeneous acceleration architectures in both academia and industry.

Overview – General‑purpose CPUs are inefficient for deep learning. CPUs have long been the universal compute core, but two factors hinder their dominance: intrinsic constraints (semiconductor process limits and memory‑bandwidth bottlenecks) and a shift in workload demand toward data‑intensive cloud computing and deep learning.

1. Intrinsic constraints. After reaching ~7 nm, semiconductor scaling approaches physical limits, weakening Moore’s law benefits. To achieve higher performance and lower power, designers increasingly specialize (e.g., GPUs, custom ASICs) rather than expanding generic cores. Moreover, CPUs suffer from limited off‑chip DDR bandwidth and high latency; on‑chip caches mitigate this but occupy a large silicon area, leaving less than 1 % of the die for actual arithmetic logic. Compatibility requirements further restrict architectural innovation.

2. Demand shift. Deep learning models (especially CNNs) grow deeper and require far more compute, while their workloads are highly parallel, data‑dense, and have high reuse. These characteristics favor architectures with massive parallelism and bandwidth rather than complex task scheduling, making CPUs unsuitable.

The article then presents a classic comparison chart positioning CPUs against mainstream heterogeneous processors (GPU, FPGA, ASIC) along axes of programmability/flexibility versus development difficulty/customization/efficiency/power.

GPUs sacrifice generality for thousands of cores, achieving higher compute density but demanding large memory bandwidth and sophisticated caching (including HBM). FPGAs and ASICs target specific applications; ASICs deliver the highest efficiency and lowest power but require extensive design effort. Because deep learning workloads evolve rapidly, many organizations now design domain‑specific AI processors—FPGA/ASIC hybrids that can handle a broad class of models (e.g., CNNs, RNNs).

AI Processor Landscape. AI processors are divided into cloud‑side and edge‑side solutions. A publicly maintained list (https://basicmi.github.io/Deep-Learning-Processor-List/) enumerates current designs. The development timeline shows two major stages:

1) IO‑bandwidth problem solving. Early AI processors (first‑gen TPU, FPGA designs, Cambricon ASICs) focused on increasing parallelism while coping with limited off‑chip bandwidth. Techniques such as higher data reuse, on‑chip caching, model quantization, and sparsification were employed.

2) Compute‑scaling problem solving. The advent of high‑bandwidth memory (HBM/HMC) lifted the bandwidth ceiling, enabling entire models to reside on‑chip (GB‑level caches) and providing >50× bandwidth over traditional DDR interfaces. This shift moves the design focus to scalable compute architectures and distributed processing capabilities for massive training workloads.

Figures in the original text illustrate bandwidth calculations for a single MAC unit, cache area consumption in Google’s first‑gen TPU and Cambricon’s DiaoNao ASIC, and the 3‑D stacking of HBM.

In summary, CPUs are increasingly unsuitable for deep‑learning and big‑data workloads due to limited parallelism and memory bandwidth. Heterogeneous solutions—GPUs, FPGAs, ASICs, and emerging AI processors with HBM—address these gaps through specialization, higher on‑chip compute density, and advanced memory technologies.

References: [1] 王逵, “CPU和GPU双低效，摩尔定律之后一万倍 ——写于TPU版AlphaGo重出江湖之际”, 新智元, 2017. [2] Jeff Dean, "Keynote: Recent Advances in Artificial Intelligence via Machine Learning and the Implications for Computer System Design", Hotchips 2017, 2017.