Unlocking NVMe: A Deep Dive into PCIe, Registers, and Command Architecture

Overview

NVMe (Non‑Volatile Memory Express) is a logical device interface specification similar to AHCI, defining a bus‑level protocol for accessing non‑volatile storage media (e.g., flash‑based SSDs) attached via PCI Express (PCIe).

1. Specification Overview

1.1 Terminology

Namespace

A Namespace is a collection of logical blocks (LB) whose attributes are defined in the Identify Controller data structure.

Fused Operations

Fused Operations aggregate two commands that must remain adjacent in the queue; only NVM commands support this, and atomicity of the two commands must be guaranteed.

Command Execution Order

Except for fused operations, each command in a Submission Queue (SQ) is independent; data consistency issues (e.g., RAW) are the host’s responsibility.

Write Atomicity

The controller must support atomic write units, but the host can configure the Write Atomicity feature to reduce the atomic unit size for performance.

Metadata

Optional extra information that can provide verification functions.

Arbitration Mechanism

Three arbitration methods select the next SQ to execute: round‑robin (RR), weighted RR (four priority levels), and vendor‑specific custom implementations.

Logical Block (LB)

NVMe defines the smallest read/write unit (e.g., 2 KB, 4 KB) addressed by LBA; a contiguous range of LBs forms an LBA range.

Queue Pair

Consists of a Submission Queue (SQ) and a Completion Queue (CQ); the host submits commands via SQ, and the controller returns completions via CQ.

NVM Subsystem

Includes the controller, NVM storage media, and the interface between them.

2. NVMe SSD Architecture

An NVMe SSD comprises three parts: the host‑side driver (integrated in Linux, Windows, etc.), the PCIe + NVMe controller, and the FTL + NAND flash storage media.

3. PCIe Register Configuration

NVMe over PCIe abstracts the physical layer, providing an application‑layer protocol on top of PCIe. The PCIe bus has three layers (physical, data link, transport) and NVMe resides in the transport (application) layer.

3.1 PCI Header

NVMe controllers are endpoint devices (type 0) with a 64 KB PCI header.

3.2 PCI Capabilities

Configure power management, interrupt handling (MSI, MSI‑X), and PCIe capabilities.

3.3 PCI Express Extended Capabilities

Configure advanced features such as error recovery.

4. NVMe Register Definitions

Registers are split into controller‑wide attributes and per‑queue head/tail doorbell registers.

CAP – controller capabilities (page size, supported I/O commands, arbitration, etc.).

VS – version number of the NVMe specification implemented.

INTMS – interrupt mask (invalid when using MSI‑X).

INTMC – interrupt mask clear (invalid when using MSI‑X).

CC – controller configuration (I/O SQ/CQ element size, shutdown notification, arbitration, page size, enable).

CSTS – controller status (shutdown state, fatal error, ready).

AQA – Admin queue attributes (SQ size, CQ size).

ASQ – Admin SQ base address.

ACQ – Admin CQ base address.

Doorbell registers for each queue start at offset 0x1000.

4.1 Register Understanding

CAP lists all capabilities; CC selects a subset of those capabilities. Changing CC after a reset reconfigures the controller.

5. Memory Data Structures

5.1 SQ and CQ Details

Empty queues have one unused element; full queues keep one slot free to differentiate full vs. empty. Queue size is 16 bits (minimum 2 elements, maximum 64 K for I/O queues, 4 K for Admin queues). Each queue has a 16‑bit identifier (QID) assigned by the host, and optional priority levels (U, H, M, L).

5.2 Arbitration

RR

All queues have equal priority; the controller may select multiple commands from a queue per arbitration burst.

Weighted RR

Three strict priority levels (Priority 1 > Priority 2 > Priority 3); higher‑priority queues are serviced first (non‑preemptive).

Vendor‑Specific

Custom arbitration mechanisms defined by the vendor.

5.3 Data Addressing (PRP and SGL)

PRP

Physical Region Page (PRP) pointers are 64‑bit physical addresses aligned to 4 KB pages. Two addressing modes exist: direct PRP pointer or PRP List for multi‑page transfers. Admin commands must use PRP; I/O commands may use PRP or SGL.

SGL

Scatter‑Gather List (SGL) consists of segments, each containing descriptors that point to arbitrary memory regions. Six descriptor types are defined (data, garbage, segment, last‑segment, keyed data, transport data).

PRP vs. SGL Comparison

Both describe memory regions; PRP maps to physical pages, while SGL can describe any contiguous physical space, offering greater flexibility.

6. NVMe Command Set

6.1 Command Execution Flow

The host writes commands to an SQ, updates the tail doorbell, the controller DMA‑fetches the command, executes it, writes a completion to the CQ, generates an interrupt (typically MSI‑X), and the host processes the completion and updates the head doorbell.

6.2 Command Classification

Commands are divided into Admin (controller management) and NVM (I/O) commands. Admin commands are submitted to the Admin queue pair; NVM commands to I/O queue pairs. Each command is 64 bytes with a common format.

6.3 Admin Commands (selected)

00h – Delete I/O SQ.

01h – Create I/O SQ.

02h – Get Log Page.

04h – Delete I/O CQ.

05h – Create I/O CQ.

06h – Identify (returns controller and namespace data structures).

08h – Abort (best‑effort command cancellation).

09h – Set Features.

0Ah – Get Features.

0Ch – Asynchronous Event Request.

10h – Firmware Activate.

11h – Firmware Image Download.

6.4 NVM Commands (selected)

00h – Flush (commit data and metadata).

01h – Write.

02h – Read.

04h – Write Uncorrectable (mark data block as invalid).

05h – Compare (compare host buffer with data from NVM).

09h – Dataset Management (hint usage patterns for performance).

7. Controller Structure and Lifecycle

7.1 Controller Structure

Three functional categories: I/O, Admin, and Discovery. Typically a single Admin controller manages the device.

7.2 Command Execution Process

Host writes command(s) to SQ.

Host updates SQ tail doorbell.

Controller fetches command via DMA.

Controller executes command.

Controller writes completion to CQ.

Controller notifies host via interrupt.

Host processes completion and updates CQ head doorbell.

7.3 Reset

Controller reset (triggered by PCIe reset, power cycle, or CC.EN = 0) deletes all I/O queues, aborts pending commands, clears CSTS.RDY, but preserves AQA, ASQ, ACQ. After reset, the host re‑enables the controller, configures registers, and recreates queues.

7.4 Interrupts

NVMe supports pin‑based, single MSI, multi‑message MSI, and MSI‑X; MSI‑X is recommended for up to 2 K vectors, allowing each CQ to generate its own interrupt.

7.5 Controller Initialization

Configure PCI and PCIe registers.

Wait for CSTS.RDY to become set.

Configure AQA, ASQ, ACQ.

Configure CC.

Set CC.EN = 1.

Wait for CSTS.RDY = 1.

Issue Identify to discover controller and namespace structures.

Get features to learn I/O queue capabilities and configure interrupts.

Allocate I/O CQ and SQ.

Optionally submit Asynchronous Event Request for health monitoring.

7.6 Firmware Update

Download firmware image using Firmware Image Download command.

Activate firmware with Firmware Activate command.

Controller resets.

Host re‑initializes the controller and queues.

7.7 Metadata Transfer

Metadata can be appended to a logical block or transferred as a separate logical block, providing protection information such as CRC, application tag, and reference tag.

7.8 End‑to‑End Data Protection

Data protection uses metadata (Guard, Application Tag, Reference Tag) to detect errors on the PCIe link and within the flash media. Four protection scenarios exist, but the protocol defines three based on the presence of protection and the PRACT bit.