Breaking the Storage Wall: How In‑Memory Computing Is Shaping AI Chip Design

1. Compute Architecture and Storage Bottlenecks

Rapid growth in AI workloads has exposed power, performance, memory, and Moore’s‑law slowdown limits in existing architectures, prompting a need for innovations that break both compute and storage walls.

2. Paths to Overcome the Storage Wall

High‑bandwidth data communication : Serial SerDes links, optical interconnects, and 2.5D/3D stacking increase bandwidth without changing the underlying process.

Data‑near‑compute : Adding more cache levels and employing high‑density on‑chip memories such as EDRAM or PCM bring data closer to the processor.

In‑memory computing (Compute‑in‑Memory) : Near‑data processing and tightly integrated compute‑storage (e.g., HBM, 3D‑X‑stacking) embed arithmetic directly in memory arrays.

3. Principles, Advantages, and Applications of Compute‑in‑Memory

The classic von Neumann architecture separates CPU and memory, causing costly data movement. Compute‑in‑memory merges arithmetic with storage, reducing data transfer latency and energy. Because the approach relies on analog or low‑precision (≈8‑bit) operations, it suits AI inference and edge‑device workloads where high precision is less critical.

Typical use cases include IoT devices with built‑in flash, embedded AI that tolerates lower precision, and scenarios where large on‑chip storage is already required.

4. Types and Trade‑offs

Off‑chip storage integration (e.g., HBM with GPUs) offers very high bandwidth (≈900 GB/s) but at high power and cost. SRAM alternatives lower energy but increase area.

Emerging memory technologies such as MRAM provide non‑volatile storage with high density and low power, achieving around 10 TOPS/W.

On‑chip storage integration embeds algorithms (weights) directly in memory cells, using technologies like PCM, RRAM/Memristor, and floating‑gate devices. These enable parallel MAC operations with configurable precision, making them attractive for deep‑learning inference.

5. Chip Optimization Strategies

Edge inference chips prioritize low cost, low power, and relaxed precision.

Cloud‑side training chips require higher precision and generality; current compute‑in‑memory solutions are more suited to embedded front‑end workloads.

6. Challenges

Floating‑gate memory is not yet optimized for arithmetic.

New memory devices must evolve to better support compute‑in‑memory.

Current precision is limited to 8 bit; higher‑bit implementations (e.g., 10 bit) are needed for broader adoption.

Compatibility with existing design flows, tools, and process nodes requires ecosystem development.

Real‑world performance must be demonstrated in target applications.

7. Future Outlook

Advances in low‑precision MAC units promise >300 TOPS/W, far exceeding today’s 5‑50 TOPS/W. Continued scaling of floating‑gate and emerging memories (from 40 nm toward 14 nm and 28 nm to 5 nm) will boost efficiency and density.

8. Key Applications

Power‑efficient edge devices such as smart home hubs, wearables, low‑power IoT sensors, and city‑scale edge compute platforms benefit from compute‑in‑memory architectures.

9. Important Players

IBM : Demonstrated neural‑network inference using PCM‑based compute‑in‑memory.

Prof. Xie Yuan (UC Santa Barbara) : Developed the PRIME architecture on ReRAM, achieving 20× power reduction and 50× speedup.

Syntiant (USA) : Built ultra‑low‑power Neural Decision Processors targeting audio inference, claiming ~20 TOPS/W.

Mythic (USA) : Uses analog flash arrays for MAC operations, targeting ~0.5 J per MAC (≈40 TOPS/W).

ZhiCun Technology (China) : First domestic compute‑in‑memory chip focused on ultra‑low‑power voice recognition.

NewMem Technology (China) : R&D on memristor‑based compute‑in‑memory, backed by Tsinghua University.