TPU vs NPU: How Specialized AI Chips Outperform GPUs

GPUs dominate AI but were originally designed for graphics rendering. AI workloads have unique demands: matrix multiplication and convolution are core operations; low-precision arithmetic (INT8, FP16) often suffices; inference prioritizes energy efficiency over raw performance; and specific model structures can be optimized at the hardware level. Purpose-built AI chips address these needs.

TPU: Google's Tensor Processing Unit

What Is a TPU?

TPU (Tensor Processing Unit) is Google's custom AI accelerator designed specifically for TensorFlow and machine learning tasks.

Architecture: Systolic Array

The core of a TPU is a systolic array . Unlike a GPU where each core computes independently and moves data frequently, data in a systolic array flows between compute units, reducing data movement overhead.

Standard GPU: each core computes independently, heavy data movement
TPU: data "flows" between compute units, reducing movement

Input data → [Compute Unit] → [Compute Unit] → [Compute Unit] → Output
              ↑                ↑                ↑
         Weight data flows in from above

This resembles an assembly line: data moves like parts on a conveyor belt, processed at each station, minimizing back-and-forth transfers.

Generational Evolution

TPU v1 (2016) : Inference-only, INT8 compute

TPU v2 (2017) : Supports training, BF16 precision

TPU v3 (2018) : Liquid cooling, major performance boost

TPU v4 (2021) : Over 2× performance vs v3

TPU v5p (2023) : Larger-scale clusters

TPU v6e (Trillium) (2024) : Latest generation

How TPUs Are Used

Power Google Search, Translate, Photos, and other services.

Available for rent on Google Cloud (not sold directly).

Primarily paired with TensorFlow and JAX frameworks.

NPU: Neural Processing Unit

What Is an NPU?

NPU (Neural Processing Unit) is a class of processors designed specifically for neural network computation, typically integrated into a System-on-Chip (SoC).

Where NPUs Are Found

Huawei Kirin/Ascend : Da Vinci architecture NPU.

Apple A/M series : Neural Engine.

Qualcomm Snapdragon : Hexagon NPU.

MediaTek Dimensity : APU (AI Processing Unit).

Intel Core Ultra : Integrated NPU.

Typical On-Device NPU Tasks

Photo enhancement : Scene detection, AI beautification, night mode

Face recognition : Face ID, unlock

Voice assistants : Speech recognition, natural language processing

Real-time translation : On-device translation without network

Video effects : AI segmentation, background blur

Text recognition : OCR, document scanning

NPU vs GPU for AI

Design goal : NPU is AI-specific; GPU is general parallel compute

Energy efficiency : NPU higher; GPU lower

Absolute performance : NPU suited for edge inference; GPU stronger, suited for training

Integration : NPU embedded in SoC; GPU discrete or integrated

Typical use case : NPU for mobile/PC on-device AI; GPU for training and large-scale inference

NPU advantages:

Low power consumption – battery-friendly for phones.

Low latency – local processing avoids network round-trips.

Privacy – data stays on device, not uploaded to cloud.

Other Notable AI Accelerators

Huawei Ascend

Ascend 310 : Inference chip, low power.

Ascend 910/910B : Training chips, benchmarked against NVIDIA A100.

Used with MindSpore framework, forming a domestic Chinese AI compute ecosystem.

Cambricon

MLU (Siyuan) series for data-center AI acceleration.

Terminal IP licensed to chip vendors for integration.

Paired with proprietary software stack.

Google Edge TPU

Miniaturized TPU for edge devices.

USB accelerator (e.g., for Raspberry Pi).

Runs local inference with TensorFlow Lite.

Intel Habana (Gaudi)

Acquired by Intel; Gaudi series targets data-center AI training.

Noted for competitive price-performance.

ASIC vs FPGA

ASIC (Application-Specific Integrated Circuit)

TPU is an ASIC.

Pros: Highest efficiency, lowest power.

Cons: Fixed function after tape-out; high development cost and long cycle.

Best for: Well-defined, high-volume AI workloads.

FPGA (Field-Programmable Gate Array)

Reconfigurable "semi-custom" silicon.

Pros: Flexibility to adapt to algorithm changes.

Cons: Lower efficiency than ASIC.

Best for: Evolving algorithms, need for adaptability.

Key vendors: Intel Altera, Xilinx (now AMD).

Key Metrics for AI Chips

TOPS (Trillion Operations Per Second)

Common measure of AI compute throughput (typically INT8):

iPhone 15 Pro NPU: ~35 TOPS

Snapdragon 8 Gen 3 NPU: ~70 TOPS

Apple M4 Neural Engine: ~38 TOPS

NVIDIA RTX 4090: ~330 TOPS

NVIDIA H100: ~3,958 TOPS

Huawei Ascend 910B: ~320 TOPS

Energy Efficiency (TOPS/W)

Edge NPUs generally achieve higher TOPS/W due to lower precision and smaller scale.

Data-center chips prioritize absolute performance over efficiency.

Model Compatibility

Ability to run models from mainstream frameworks (PyTorch, TensorFlow) is critical for real-world deployment.

Market Landscape

Training

NVIDIA GPUs (H100/B200) hold dominant share.

Huawei Ascend 910 series is the primary alternative in China.

Google TPU mainly serves internal and cloud customers.

Inference

NVIDIA GPUs remain mainstream.

Specialized ASICs emerging (Groq, Cerebras, etc.).

Edge NPUs rapidly proliferating.

Edge

Nearly all smartphone SoCs now integrate an NPU.

PCs adopting NPUs (Intel Core Ultra, Snapdragon X Elite).

"AI PC" and "AI Phone" become marketing differentiators.

Conclusion

GPU is the "king of general parallel computing," but on the specific battlefield of AI, dedicated accelerators are rising. TPU: Google-built, born for TensorFlow. NPU: The AI accelerator in your phone, enabling low-power on-device intelligence. Ascend: China's sovereign AI compute foundation. ASIC: Ultimate efficiency, zero flexibility. The future of AI compute is likely heterogeneous – training on GPU/TPU, cloud inference on ASIC, edge inference on NPU – each playing its role.