How PCIe Devices Are Discovered and Accessed in x86 Systems

PCIe Topology and Enumeration

PCI Express (PCIe) supports up to 256 buses. Each bus can host up to 32 devices, and each device can provide up to 8 functions, giving every endpoint a unique BDF (Bus, Device, Function) identifier.

Architecture Components

A typical x86 PCIe hierarchy consists of a Root Complex, optional Switches, and Endpoint devices (including embedded endpoints that lack external connectors). After power‑on the host must discover and configure every component.

Depth‑First Enumeration Process

The host traverses the PCIe tree depth‑first, reading configuration space to identify each node and writing configuration registers to assign bus numbers. The algorithm ensures each node is visited once.

Start at Bus 0 (Root Complex). Ignore embedded endpoints, discover Bridge 1, assign it Bus 1, set Primary = 0, Secondary = 1, Subordinate = 0xFF (temporary).

Scan Bus 1, find Bridge 3 (a switch). Assign it Bus 2, set Primary = 1, Secondary = 2, Subordinate = 0xFF.

Scan Bus 2, discover Bridge 4 and the NVMe SSD endpoint. Assign Bridge 4 Bus 3, set Primary = 2, Secondary = 3, Subordinate = 3 (leaf).

Continue Bus 2, discover Bridge 5 and a NIC endpoint. Assign Bridge 5 Bus 4, set Primary = 2, Secondary = 4, Subordinate = 4.

After completing Buses 3 and 4, set Bridge 3’s Subordinate = 4, then return to Bridge 1 and set its Subordinate = 4.

Return to Bus 0, discover Bridge 2 (graphics card). Assign it Bus 5, set Primary = 0, Secondary = 5, Subordinate = 5.

When enumeration finishes, the host possesses a complete PCIe topology.

Inspecting the Topology on Linux

Use the command: lspci -v -t which prints a tree of discovered devices. In the example the STAR1000 NVMe SSD appears at BDF 3c:00.0 with upstream port 00:1d.0.

Reading PCIe Configuration Space

Dump the raw configuration registers of the SSD with: lspci -xxx -s 3c:00.0 The dump shows:

Vendor ID and Device ID.

Class code 0x010802 (NVMe storage controller).

Capability pointer at offset 0x40. Following the pointer reveals capabilities such as Power Management, MSI, Link Control & Status, MSI‑X.

Link width field 0x43 indicates a x4 lane link supporting PCIe Gen3 (8 Gbps per lane).

PCIe Memory Space and BARs

Each endpoint is allocated one or more Base Address Registers (BARs) that map device memory into the CPU address space. The STAR1000 SSD exposes two BARs:

BAR0: 1 MiB region.

BAR1: 256 KiB region.

The host programs base addresses for these regions, enabling memory‑mapped I/O to the SSD.

NVMe Command Example Using PCIe Memory Access

1. Doorbell write (PCIe Memory Write) – The host writes to the SSD’s doorbell register to submit a command.

2. NVMe read request (PCIe Memory Read) – The SSD issues a Memory Read to fetch data; the host returns a Completion with Data (CplD) packet.

All NVMe operations—queue management, command submission, data transfer—are performed via such PCIe memory transactions.

Evolution of PCIe Generations

PCIe continues to evolve:

PCIe Gen4 (released 2017) provides 16 Gbps per lane.

PCIe Gen5 (under development) targets 32 Gbps per lane.

Understanding enumeration and memory‑mapped access is essential for designing, debugging, and optimizing modern storage and networking solutions.