Unveiling CPU Cache: How It Bridges the Speed Gap Between CPU and Memory

Computer Performance Bottleneck

In the von Neumann architecture, memory is organized in a hierarchy: registers, cache, main memory (RAM), SSD, HDD, and network storage. The closer a storage component is to the CPU, the faster its access time but the smaller its capacity and the higher its cost per byte.

For example, a 3.0 GHz CPU can access a register in a single clock cycle (~0.3 ns), while main memory takes about 120 ns, an SSD 50‑150 µs, a mechanical HDD 1‑10 ms, and network access tens of milliseconds. If we exaggerate a clock cycle to one second, the relative access times become: registers ~1 s, RAM ~6 minutes, SSD ~2‑6 days, HDD ~1‑12 months, network access years.

DRAM is dynamic random‑access memory. Image source: "How L1 and L2 CPU Caches Work, and Why They're an Essential Part of Modern Chips".

Cache Development History

CPU cache uses SRAM (Static Random‑Access Memory) chips, which retain data while powered but lose it when power is removed. Modern CPUs typically have three levels of cache: L1, L2, and L3.

Early CPUs (pre‑80286) had no cache; the CPU accessed memory directly. Starting with the 80386, the mismatch between CPU speed and memory speed became evident, prompting the introduction of external cache.

The 80486 integrated an 8 KB L1 Cache on the die and supported an external L2 Cache (128‑256 KB). Although the L1 size was modest, it was sufficient for the CPUs of that era.

Image source: "How L1 and L2 CPU Caches Work, and Why They're an Essential Part of Modern Chips".

Increasing the L1 size yielded diminishing returns, while enlarging L2 dramatically improved overall hit rates, making larger but slower L2 caches a cost‑effective choice.

With the Pentium‑1 era, CPUs adopted a superscalar design with separate instruction and data L1 caches (each 8 KB) and an external L2 cache. The Pentium Pro moved the L2 cache onto the die, establishing the modern three‑level cache hierarchy.

In the multi‑core era, Intel introduced per‑core L1/L2 caches with a shared L3 cache (Smart Cache). Modern CPUs may also feature an L4 cache, and future designs could add more levels.

How Cache Bridges the CPU‑Memory Performance Gap

Cache exploits the principle of locality—both spatial and temporal—to keep recently accessed data and instructions close to the CPU, reducing the number of slow main‑memory accesses.

The two forms of locality are:

Temporal locality: Memory locations that have been accessed recently are likely to be accessed again soon.

Spatial locality: Memory locations near a recently accessed address are likely to be accessed in the near future.

When the CPU needs data, it first checks the L1 Cache. A hit returns the data instantly. If missed, the CPU probes L2 Cache, then L3 Cache, and finally main memory. By storing both the requested line and its neighboring lines, the cache maximizes the chance of future hits.

Conclusion

This article introduced the performance bottleneck caused by the CPU‑memory speed gap, traced the evolution of CPU cache from non‑existent to multi‑level hierarchies, and explained how locality principles enable caches to dramatically improve overall system performance. Future developments may add deeper levels such as L4 and beyond.