Why GPUs Lag Behind Big AI Models and How In‑Memory Computing Helps

1. Bottlenecks of Large‑Scale AI Model Computation

AI models have grown from linear to exponential size (e.g., GPT‑3, AlphaFold2). Traditional acceleration methods such as larger instruction sets, prefetching, SIMD/SIMT, cache compression, and higher parallelism have been used for years but do not fundamentally solve data‑intensive workloads.

Storage‑Compute Separation Bandwidth

The primary bottleneck is the limited bandwidth between separate memory and compute units. In the post‑Moore era, memory bandwidth caps the effective compute bandwidth, illustrated by training BERT on eight 1080 Ti GPUs taking 99 days.

Energy Density and Efficiency

Separating memory and compute also inflates energy consumption; data movement can consume 60‑90 % of total power, creating a “memory wall” that hinders both performance and efficiency.

GPU Architecture Evolution

Modern GPUs allocate an increasing fraction of die area to memory, shifting from compute‑centric to data‑flow‑centric designs, yet data‑transfer power remains a dominant constraint, especially for exascale supercomputers where >50 % of power is spent on data movement.

2. Advantages and Design Challenges of Compute‑In‑Memory (CIM)

Integrating compute capability directly into memory cells can deliver >1000 TOPS, energy efficiency of 10‑100 TOPS/W, and cost reductions by an order of magnitude.

Reduced data movement (energy cut by 10‑100×).

Memory cells act as compute units, scaling compute density without enlarging chip area.

Single CIM unit replaces separate logic and registers, yielding smaller, faster blocks.

Technical Routes

Processing‑In‑Memory (PIM) : Look‑up table computation inside memory, already deployed in GPUs.

Near‑Memory Computing : Compute modules placed close to memory, exemplified by AMD’s Zen CPUs.

In‑Memory Computing : Dedicated compute cores inside memory chips (e.g., Mythic, Qi‑Chip, Flash‑AI), suitable for fixed‑algorithm workloads.

Logic‑In‑Memory : Embedding logic within memory arrays, demonstrated by TSMC and Qi‑Chip, offering shortest data paths and high precision for large models.

Application Domains

CIM can accelerate personalized recommendation, speech recognition, natural language processing, autonomous driving, and industrial vision, enhancing both cloud‑scale and edge devices.

Overall, moving toward storage‑compute integration and CIM technologies is essential to break the bandwidth and energy walls that limit current GPU‑centric AI training, enabling scalable, high‑efficiency compute for both cloud and edge scenarios.