Understanding Instruction Pipelines and Hazards in CPU Architecture

I am "A Q", an employee in CPU workshop 1, which has eight cores, each core representing a workshop that executes machine instructions compiled from high‑level code.

When a leader questions why I appear idle, I explain that my current instruction is waiting for data from memory, so the fetch unit is idle and can start fetching the next instruction – not idling, just waiting for slower memory.

Instruction Pipeline

Leadership realizes that our CPU wastes resources by processing one instruction at a time. By overlapping the four classic stages—fetch, decode, execute, and write‑back—different stages can use freed‑up hardware to work on subsequent instructions, forming an instruction pipeline.

After adopting a four‑stage pipeline, performance improves dramatically, but competitors soon copy the technique, prompting us to increase pipeline depth.

Pipeline Depth

We double the pipeline stages from four to eight, halving the time per stage. This further reduces execution time (e.g., four instructions finish in 7 ns instead of 16 ns) but also adds more hardware and power consumption.

When competitors increase their depth even more, we engage in a "pipeline depth war," eventually reaching thirty stages, after which diminishing returns and higher power usage force us to find a balanced depth.

Hazards in the Pipeline

While pipelines boost throughput, they introduce three kinds of hazards:

Structural Hazard : hardware resources are contested (e.g., fetch and load units both need memory at the same time). Data Hazard : a later instruction depends on the result of an earlier one that has not yet completed. Control Hazard : the next instruction to fetch depends on the outcome of a branch instruction.

These hazards cause pipeline stalls, reducing the very performance gains we sought. The story ends with leaders pondering new strategies to mitigate these issues.