Why GPUs May Lose the AI Race: TPU, FPGA, and Future Hardware Trends

Background

Artificial‑intelligence software algorithms—such as CNN, RNN, and DNN—are large‑scale parallel‑compute workloads that historically rely on graphics processing units (GPUs) for acceleration. Early AI projects used a variety of parallel chips (GPU, FPGA, ASIC), but GPUs became the most mature and widely deployed solution, powering Google image‑recognition, Tesla and Volvo autonomous‑driving projects.

GPU Advantages and Limitations

Nvidia’s 2016 quarterly report showed that data‑center and automotive GPU revenue, though small compared with PC gaming, grew at a 63 % year‑over‑year rate, indicating strong momentum. Nvidia also introduced the Pascal platform and its own AI algorithms, reinforcing the perception that GPUs dominate AI acceleration. However, the article questions whether GPUs will remain the sole hardware accelerator for AI, noting several inherent drawbacks.

Google’s Tensor Processing Unit (TPU)

At the 2016 I/O conference Google unveiled the Tensor Processing Unit (TPU), a custom ASIC optimized for TensorFlow. The TPU’s design philosophy is to cut precision for AI workloads, achieving roughly an order‑of‑magnitude improvement in energy efficiency over traditional GPU‑based acceleration. By reducing the floating‑point word width from the IEEE‑754 32‑bit standard to as low as 8 bits for certain operations, the TPU dramatically shrinks transistor count and power consumption in its arithmetic units.

The article highlights that the GPU’s largest compute block, the ALU, contains a multiply‑add (MA) unit whose latency scales with log₂(N) where N is the word width. Reducing the word width from 24 bits to 8 bits can cut the ALU area to about 1/14 of its original size, offering a ten‑fold improvement in area and power.

GPU vs. FPGA for AI Acceleration

GPU compute units are designed for high‑precision image processing; this excess precision wastes energy for many AI tasks.

FPGA lookup‑table (LUT) resources are weak for low‑precision floating‑point operations and lack dedicated AI optimizations.

Both GPU and FPGA architectures were originally built for workloads that differ significantly from neural‑network computation, leading to mismatched on‑chip network (NOC) designs.

Benchmark data from Auviz Systems (2015) shows a high‑end FPGA can process 14 images / second / watt, whereas a comparable GPU handles only 4 images / second / watt, illustrating the potential efficiency advantage of FPGA when the NOC is properly leveraged.

On‑Chip Network (NOC) Challenges

GPU’s SIMT (single‑instruction‑multiple‑thread) model relies on a simple shared‑memory communication scheme: each compute node writes to a shared address and later reads it, which becomes a bottleneck when communication volume increases. In contrast, FPGA designs use a mesh‑style NOC with programmable routing, allowing direct node‑to‑node transfers and higher bandwidth, reducing contention and latency.

Both architectures therefore suffer from sub‑optimal NOC designs for neural‑network workloads, but FPGA’s flexible routing can mitigate these issues more effectively than GPU’s fixed shared‑memory approach.

Future Scenarios Where GPUs May Still Excel

VR/AR : Low‑latency rendering (<20 ms) is critical to avoid motion sickness. High‑end GPUs (e.g., Nvidia GTX 970, AMD R9 290) currently provide the necessary graphics performance, and the growth of standalone VR/AR headsets will drive demand for powerful mobile GPUs.

Cloud Computing + Gaming : Services like Amazon EC2 offer Nvidia Tesla GPUs for large‑scale floating‑point workloads. Combining GPU acceleration with cloud infrastructure enables on‑demand AI training and inference, though network limitations still constrain massive scaling.

Military Applications : Unlike consumer GPUs that prioritize peak visual performance, military GPUs require high reliability, radiation tolerance, and ruggedness, leading to divergent design priorities.

As AI moves from research to production, the pressure on existing GPU/FPGA solutions will increase, and customized NOC designs or alternative accelerators may become viable replacements.