Unlocking CPU Speed: How Cache Bridges the Gap Between Processor and Memory

In the world of computers, the CPU is like a fast rabbit while memory behaves like a slow turtle, creating a huge speed gap that hampers overall system efficiency.

To bridge this gap, engineers introduced CPU cache—a high‑speed buffer between the processor and memory that stores frequently accessed data and instructions, dramatically reducing access latency.

1. CPU Cache Overview

CPU speeds double roughly every 18 months (Moore's law), whereas memory improves far more slowly, widening the performance disparity. Modern CPUs therefore incorporate a multi‑level cache hierarchy (L1, L2, L3) where each level is larger but slower. A cache line, typically 64 bytes, is the smallest unit transferred between memory and cache.

1.1 Definition and Role

Cache acts as a temporary storage “relay station” that speeds up data reads by keeping likely‑needed data close to the CPU, avoiding costly memory accesses.

1.2 Cache Levels

L1 Cache is closest to the core, split into data (L1 D) and instruction (L1 I) caches, with capacities of a few tens of kilobytes.

L2 Cache is larger (hundreds of KB to a few MB) and serves as a backup when L1 misses.

L3 Cache is the largest (several MB to tens of MB) and is shared among cores.

Data is looked up sequentially in L1 → L2 → L3; only if all miss does the CPU fetch from main memory.

2. Core Principles of CPU Cache

2.1 Locality Principle

Cache performance relies on temporal locality (recently accessed data is likely to be accessed again) and spatial locality (data near accessed addresses is also likely to be needed).

int sum = 0;
int arr[100] = {1, 2, 3, ... , 100};
for (int i = 0; i < 100; i++) {
    sum += arr[i];
}

In this loop, the variable sum and successive array elements benefit from both types of locality, allowing the CPU to keep them in cache.

2.2 Cache Hit and Miss

A cache hit means the requested data is already in cache, enabling fast access; a miss forces a slower memory read and may trigger replacement algorithms such as LRU.

2.3 Cache Line

A cache line is the minimum data block transferred between memory and cache (commonly 64 bytes). Accessing one element of an array brings the entire line into cache, improving subsequent accesses to neighboring elements.

3. CPU Cache Write Policies

3.1 Write‑Through

Every write updates both cache and main memory, ensuring data consistency but incurring higher latency and bus traffic.

3.2 Write‑Back

Writes modify only the cache; the modified (dirty) line is written back to memory only when it is evicted, reducing memory traffic but requiring a dirty‑bit mechanism and careful handling of power‑loss scenarios.

4. Multi‑Core Cache Coherence

4.1 The Coherence Problem

When multiple cores have private caches, a write by one core can leave other cores with stale copies, leading to incorrect results.

4.2 Solutions

Bus Snooping lets each core monitor the bus for writes to shared addresses, invalidating or updating its own cache lines accordingly.

MESI Protocol extends snooping with four states—Modified (M), Exclusive (E), Shared (S), Invalid (I)—to reduce unnecessary traffic.

M : Modified, only this core has the latest data.

E : Exclusive, data matches memory and no other core holds it.

S : Shared, multiple cores hold identical copies.

I : Invalid, the line must be fetched again.

When a core wants to write a line in state S, it sends an invalidate message, causing others to transition to I, then moves to M.

5. Optimizing Code for CPU Cache

5.1 Data Alignment

Aligning data structures to their natural word boundaries (e.g., 4‑byte alignment for int) ensures that consecutive elements reside within the same cache line, improving hit rates.

#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>
#include <time.h>

// Unaligned structure
struct UnalignedStruct {
    char a;
    int b;
    char c;
};

// Aligned structure (4‑byte alignment)
struct __attribute__((aligned(4))) AlignedStruct {
    char a;
    int b;
    char c;
};

void testAccessTime(void *arr, size_t numElements, size_t structSize) {
    clock_t start, end;
    double cpu_time_used;
    int i;
    start = clock();
    for (i = 0; i < numElements; i++) {
        char *ptr = (char *)arr + i * structSize;
        char temp = *((char *)ptr);
        temp = *((int *)(ptr + 1));
        temp = *((char *)(ptr + 5));
    }
    end = clock();
    cpu_time_used = ((double) (end - start)) / CLOCKS_PER_SEC;
    printf("Access time: %f seconds
", cpu_time_used);
}

int main() {
    size_t numElements = 1000000;
    struct UnalignedStruct *unalignedArr = (struct UnalignedStruct *)malloc(numElements * sizeof(struct UnalignedStruct));
    struct AlignedStruct *alignedArr = (struct AlignedStruct *)malloc(numElements * sizeof(struct AlignedStruct));
    if (unalignedArr == NULL || alignedArr == NULL) {
        perror("malloc failed");
        return 1;
    }
    printf("Testing unaligned struct:
");
    testAccessTime(unalignedArr, numElements, sizeof(struct UnalignedStruct));
    printf("Testing aligned struct:
");
    testAccessTime(alignedArr, numElements, sizeof(struct AlignedStruct));
    free(unalignedArr);
    free(alignedArr);
    return 0;
}

This program shows that aligned structures achieve noticeably lower access times.

5.2 Loop Optimisation

Reducing loop nesting depth and arranging loops to follow the memory layout (row‑major for C arrays) improves spatial locality and cache hit rates.

for (int i = 0; i < ROWS; i++) {
    for (int j = 0; j < COLS; j++) {
        sum += matrix[i][j];
    }
}

Accessing elements row‑wise keeps successive accesses within the same cache line, whereas column‑wise traversal often crosses cache lines, causing more misses.