SRAM vs DRAM vs HBM: A Complete Memory Hierarchy for the AI Era

One‑Sentence Core Conclusion

SRAM is the "pocket wallet" (nanosecond, KB‑MB), DRAM the "backpack" (hundreds of nanoseconds, GB‑TB), and HBM the "high‑performance backpack" (hundreds of nanoseconds, GB with TB/s bandwidth). They form a memory pyramid, not competitors.

HBM Is Essentially a High‑End DRAM

HBM is a specially packaged DRAM built with 3D stacking and TSV (through‑silicon‑via) technology, belonging to the same DRAM family as DDR5, LPDDR5X and GDDR7.

1. Core Technology Comparison

Dimension          SRAM                         DRAM                         HBM
--------------------------------------------------------------------------
Full name          Static Random Access Memory  Dynamic Random Access Memory  High Bandwidth Memory
Underlying structure 6‑transistor flip‑flop       1 transistor + 1 capacitor   3D‑stacked DRAM + TSV
Speed (latency)   Fastest (1‑10 ns)            Medium (~80‑120 ns)          Fast (~100‑220 ns)
Bandwidth         ~19 TB/s (on‑chip)           DDR5: ~50‑80 GB/s            HBM3E: ~1.2 TB/s/stack; HBM4: ~1.6 TB/s
Capacity (per chip) KB‑MB (min)                GB‑TB (large)                24‑64 GB per stack (large)
Cost per MB        Highest (~$15‑20/MB)          Lowest                       Mid‑high (far above DDR, far below SRAM)
Power consumption  High static power (continuous)  Medium (needs refresh)  Low per‑bandwidth power
Refresh required   No                           Yes (every few tens of ms)  Yes (DRAM nature)
Typical use        CPU L1/L2/L3 cache, Groq LPU  PC/Server memory, mobile LPDDR  AI GPUs, HPC accelerators

2. Memory Hierarchy in GPUs/AI Chips (H100 Example)

┌─────────────────────────────────────────┐
│  Register                │  ~19 TB/s, 256 KB/SM │
│  ↓
│  L1 Cache / Shared SRAM │  ~3 TB/s, 228 KB/SM   │
│  ↓
│  L2 Cache (SRAM)         │  ~3.35 TB/s, 50 MB (chip‑wide) │
│  ↓
│  HBM3 (global VRAM)      │  ~3.35 TB/s, 80 GB   │
│  ↓
│  CPU DRAM (system RAM)   │  ~50‑80 GB/s, hundreds of GB │
│  ↓
│  NVMe SSD (persistent)   │  ~12.8 GB/s, TB‑scale │
└─────────────────────────────────────────┘

Shock numbers: Register bandwidth is about 800× that of HBM, while its latency is only 1/400 of HBM’s. This explains why algorithms like FlashAttention try to keep computation within SRAM as much as possible.

3. Detailed Differences

3.1 SRAM vs DRAM: Speed vs Capacity

SRAM stores 1 bit per 6‑transistor flip‑flop, no refresh needed.

Nanosecond‑level speed, comparable to CPU core cycles.

Very low density: per‑chip capacity only 1/15‑1/20 of DRAM.

Extremely expensive: Tesla Dojo D1’s 440 MB SRAM costs roughly $9,000 (≈$15‑20/MB).

DRAM uses 1 transistor + 1 capacitor per bit and must be refreshed every few tens of milliseconds.

Medium speed (hundreds of nanoseconds), large capacity, low cost.

Dominates PCs, servers and mobile devices.

3.2 DRAM vs HBM: Containment Relationship, Not Competition

HBM is a special‑packaged, special‑architecture DRAM. Think of DRAM as a large family; HBM is its high‑end member.

Comparison Item      DDR5 (standard DRAM)          HBM
--------------------------------------------------------------------------
Bus width            64 bit                         1024 bit → 2048 bit (HBM4)
Package              DIMM slot, far from CPU       Co‑WoS (chip‑on‑chip) very close to GPU
Connection           PCB traces                     TSV + micro‑bump, vertical stack
Bandwidth            ~50‑80 GB/s                    ~1.2‑1.6 TB/s per stack
Latency              ~80‑120 ns                     ~100‑220 ns (slightly higher but acceptable)
Cost                 Low                            Very high (TSV, 3D stacking, advanced packaging)

Why HBM’s bandwidth is so high: the bus width, not frequency, is the key. Traditional DDR is like a 4‑lane highway; HBM is a 1024‑lane super‑highway, delivering orders‑of‑magnitude higher throughput even at similar per‑lane speeds.

4. Memory Roles in AI Chips

Training Scenario (Llama 70B example)

Data Type                Stored In          Reason
------------------------------------------------------------
Model weights (140 GB FP16)   HBM          Large size, high bandwidth
KV cache (dynamic)            HBM          Real‑time read/write
Gradients (training)          HBM          Large capacity needed
Optimizer state (420 GB AdamW) HBM/DRAM   Massive size
Current layer activations      SRAM (L1/L2)  Temporary compute buffers
Register operands              Register     Actual arithmetic

Core bottleneck: Training is limited by memory bandwidth; model FLOPs utilization (MFU) typically only 30‑50 %. GPUs stall on memory access rather than compute.

Inference Scenario

Small models / low‑latency needs can use a pure SRAM solution (e.g., Groq LPU) but the entire model must fit in SRAM.

Large‑model inference requires HBM + SRAM cache: HBM holds weights, SRAM buffers compute.

5. Frontier Exploration: Can SRAM Replace HBM?

Answer: Not yet, but research is ongoing

Cerebras and Groq attempts (failed): Both tried ultra‑large chips with all‑SRAM memory, but ran out of on‑chip SRAM capacity for large models and had to retrofit DRAM.

HBS (High‑Bandwidth SRAM) – concept by HBM pioneer Kim Jong‑ho:

Make an entire 12‑inch wafer of SRAM, vertically stack 12‑16 layers.

Capacity could grow from 100 GB to 1.6 TB.

Speed claimed to be 1,000× faster than DRAM.

Major challenge: power delivery and cooling – supplying thousands of amps to GPU and stacked memory is extremely difficult.

HBF (High‑Bandwidth Flash) – alternative path:

Stack flash memory like HBM to achieve near‑HBM bandwidth.

Capacity >10× HBM; example: 512 GB capacity, 1 638 GB/s read bandwidth.

Compared to HBM4’s 48 GB at the same bandwidth, HBF is suited for read‑intensive inference workloads.

6. Summary: Memory Pyramid

┌─────────┐
                │ SRAM    │ ← nanosecond, KB‑MB, most expensive
                │(L1/L2/L3)│   CPU cache, on‑chip buffer
                ├─────────┤
                │ HBM     │ ← hundred‑nanosecond, GB, TB/s bandwidth
                │(3D DRAM)│   AI GPU main memory, HPC
                ├─────────┤
                │ DRAM    │ ← hundred‑nanosecond, GB‑TB
                │(DDR/LPDDR)│ PC/server/mobile memory
                ├─────────┤
                │ NAND    │ ← microsecond, TB
                │(SSD)    │ Persistent storage, datasets
                └─────────┘

SRAM solves the "speed" problem but is limited by capacity and cost.

DRAM is the foundation of general‑purpose computing: low cost, large capacity.

HBM is a high‑end DRAM variant that uses 3D stacking and wide buses to break the AI "memory wall".

The three layers will coexist and be jointly optimized; next‑gen NVIDIA Rubin architectures will likely combine HBM4 with large‑capacity SRAM caches.