HBF High-Bandwidth Flash for LLM Inference: Simulation Reveals Benefits and Hard Technical Barriers

Background: LLM Inference Memory Wall

Large language model inference splits into Prefill (pre-filling) and Decode (decoding) phases. Model weights and context-growing KV caches create massive memory pressure. For example, Llama3 405B requires 800 GB+ memory capacity and 8 TB/s+ bandwidth . Current industry practice stacks multiple GPUs with HBM (High Bandwidth Memory), aggregating capacity and bandwidth, but this is costly and HBM bandwidth often goes underutilized.

HBF Technology Overview

SanDisk proposes HBF (High Bandwidth Flash) , borrowing HBM's 3D stacking approach but using NAND flash. The goal: approach HBM read bandwidth while delivering 8–16× HBM capacity . Samsung and SK Hynix have also entered this space. HBF is seen as a candidate to break the AI inference memory wall, yet the industry lacks systematic quantitative evaluation.

Experimental Design

The study compares three GPU memory architectures:

Pure HBM

HBM + HBF hybrid

Pure HBF

Models tested: Llama3 405B (dense) and Llama4 Maverick (MoE). Workloads: short, medium, and long real-world context lengths. Simulation tool: LLMSimulator . Service-level objective (SLO): 0.1 s TPOT (time per output token).

Core Simulation Findings (HBF Benefits)

Lowers minimum GPU hardware threshold : Under equal SLO, HBF's huge capacity lets fewer GPUs support the same model, shrinking cluster size.

Boosts system throughput (TPS) and batch size : Gains are especially pronounced in long-context scenarios; MoE models show even larger improvements .

Increases system flexibility : When SLO latency constraints are relaxed, HBF's performance advantage widens further.

Diminishing returns with more GPUs : As GPU count grows, all architectures see marginal gains taper off; HBF's value concentrates in small-to-medium clusters .

Three Hard Technical Conditions (Capacity Alone Is Not Enough)

Read bandwidth must match HBM : Conventional NAND-based HBF (CONV series) falls short on bandwidth, yielding poor performance. Only achieving HBM-class read bandwidth unlocks full benefits.

Flash endurance needs 25× improvement : KV caches generate heavy write traffic. Simulation shows current SLC program/erase cycle limits cannot sustain long-term production workloads; write endurance is a major blocker.

Programming write speed must increase : KV cache writes impose significant performance overhead; high write latency drags down decode-stage throughput.

Conclusion

HBF, leveraging NAND's ultra-high capacity, has potential to become an HBM supplement that improves LLM inference hardware cost, batch capability, and deployment barrier. It cannot simply replace HBM. Deployment requires solving three hardware-level hard problems—read bandwidth, write speed, and flash endurance—plus co-optimization at architecture and system software levels. A hybrid architecture (retain a small HBM pool for hot dynamic data; offload weights and large KV caches to HBF) is the more realistic landing path.