Breaking the Storage Wall: In‑Memory Computing and Integrated Compute‑Storage Architectures for AI

1. Compute Architecture and Storage Bottlenecks

With the surge in AI compute demand, existing architectures face power, performance, memory, and Moore's law slowdown walls, prompting innovation to break the storage wall.

2. Paths to Innovation

Key approaches include high‑bandwidth data communication (SerDes, optical interconnects, 2.5D/3D stacking), data‑near‑compute (larger caches, high‑density on‑chip memory such as EDRAM, PCM), and in‑memory compute (near‑data processing, integrated compute‑storage using HBM or 3D‑X‑stacking).

3. Principles, Advantages, and Applications of Integrated Compute‑Storage

The classic von Neumann architecture separates CPU and memory; in‑memory compute embeds arithmetic within memory cells, reducing data movement and latency. It excels in AI inference and edge IoT scenarios where lower precision (≈8‑bit) is acceptable, but is less suited for high‑precision training.

4. Types and Trade‑offs

(1) Off‑chip storage solutions

HBM offers ultra‑high bandwidth for GPUs but incurs high power and cost; SRAM alternatives lower energy but increase cost.

Emerging non‑volatile memories such as MRAM provide higher density and zero‑standby power, achieving ~10 TOPS/W.

(2) On‑chip storage solutions

Phase‑change memory (PCM) and resistive RAM (RRAM/Memristor) embed weights directly in memory, enabling analog MAC operations with high parallelism; floating‑gate devices offer mature processes and high density.

5. Chip Optimization Strategies

Edge inference chips prioritize low cost and power with modest precision, while cloud‑side training chips demand higher performance and flexibility.

6. Challenges

Current floating‑gate devices need redesign for compute, new memory technologies must overcome integration hurdles, precision is limited to 8‑bit (with efforts toward 10‑bit), and ecosystem compatibility remains a barrier.

7. Future Outlook

Advances aim for >300 TOPS/W efficiency, broader investment from firms like SoftBank, Intel, Microsoft, and government programs, and diversification of applications from voice assistants to security and smart‑city edge devices.

8. Major Players

IBM (PCM‑based neural nets), Prof. Xie Yuan’s team at UC Santa Barbara (ReRAM‑based PRIME architecture), Syntiant (ultra‑low‑power neural decision processors), Mythic (analog flash‑based AI chips), Zhichun Technology (China’s leading in‑memory AI chip maker), and XinYi Technology (memristor R&D).

References: Li Fei, "In‑Memory Computing, the Next Paradigm?" (China Electronics Daily); Wang Shaodi, "Architecture Innovation and Technical Challenges of Integrated Compute‑Storage AI Chips" (Zhichun Tech Open Course).