Unlocking PCIe: From Bus Basics to Linux Device Enumeration

Computer Bus System

A bus system enables communication between the host (CPU + Main Memory) and I/O peripherals such as network cards and disks.

Functionally, there are three bus types: data bus for transferring data, address bus for transmitting address information, and control bus for sending control signals.

From a system‑module perspective, the bus system consists of two major parts: the CPU bus (also called Front Side Bus) and the PCI/PCIe bus.

PCIe Bus

PCIe (Peripheral Component Interconnect Express) is a high‑speed serial bus used to connect NICs, disks, GPUs and other I/O devices.

PCIe Transmission Rate

PCIe measures speed in GT/s (giga‑transactions per second), not in Gbps. The effective bandwidth is calculated as bandwidth = transmission rate × encoding efficiency. For example, PCIe 2.0 uses an 8b/10b encoding, giving a per‑lane bandwidth of 5 GT/s × 8/10 = 500 MB/s; an 8‑lane device therefore achieves 4 GB/s.

Only the actual throughput matters in practice, and PCIe is backward compatible: a PCIe 4.0 device on a PCIe 3.0 motherboard will operate at PCIe 3.0 speeds.

PCIe Architecture

Unlike the shared parallel PCI bus, PCIe adopts a point‑to‑point serial architecture, giving each device a dedicated connection and exclusive bandwidth.

PCIe Root Complex (RC) : sits on the motherboard, connects the CPU bus to the CPU, main memory and the PCIe fabric.

PCIe Switch : acts as a hub, linking the RC to multiple PCIe devices and providing scalability.

PCIe Endpoint : the actual device on the bus, which can be a legacy PCI‑e endpoint (compatible with older PCI/PCI‑X) or a native PCIe endpoint.

PCI/PCI‑X Bridge : bridges legacy PCI/PCI‑X buses to the PCIe bus.

The PCIe transaction flow is: the Root Complex sends a Transaction Layer Packet (TLP) to the bus, the Switch routes it to the target device, the device replies with a TLP response, and the Switch forwards the response back to the Root Complex.

PCIe Peripherals

Interface Types : various connector standards illustrated in the accompanying diagram.

Form Factors : LP (low profile), FH (full height), HL (half length), FL (full length), SW (single width), DW (double width).

Slots : PCIe slots are shorter and have fewer pins than legacy PCI/PCI‑X slots, making them physically incompatible.

PCIe Device Enumeration Process

The PCIe bus forms a tree, so the OS uses a depth‑first traversal to enumerate devices.

The host powers on; the OS scans Bus 0 (Root Complex to Host Bridge). It discovers Bridge 1, assigns it Bus 1, and temporarily sets its Subordinate Bus Number to 0xFF.

The OS scans Bus 1, finds Bridge 3 (a PCIe Switch), assigns it Bus 2, and also sets its Subordinate Bus Number to 0xFF.

Scanning Bus 2 reveals Bridge 4 and an attached NVMe SSD, which becomes Bus 3; Bridge 4’s Subordinate Bus Number is set to 3.

Continuing on Bus 2, the OS discovers Bridge 5 with a NIC, creating Bus 4; Bridge 5’s Subordinate Bus Number is set to 4.

After Bus 4 is fully scanned, the OS returns to Bridge 3, finalizes its Subordinate Bus Number as 4, then returns to Bridge 1 and sets its Subordinate Bus Number to 4 as well.

The OS finally scans Bridge 2, creates Bus 5 for the graphics card, and sets its Subordinate Bus Number to 5.

At this point the OS has a complete PCIe topology.

PCIe BDF Numbering

Each PCIe device is identified by a Bus‑Device‑Function (BDF) tuple: Bus Number (0‑255), Device Number (0‑31), and Function Number (0‑7). For example, BDF 0000:01:00.0 refers to Bus 0, Device 1, Function 0.

PCIe BAR Addresses

Beyond BDF, the CPU needs the Base Address Register (BAR) to know the memory range a device occupies. A typical device has six BARs mapped within a 24‑byte configuration space, which the OS parses and exposes via the filesystem.

BARs are divided into regions such as Memory BAR (mapped into main memory for mmap access), I/O BAR (accessed via special I/O instructions), and MSI‑X BAR (used for interrupt vectors).

Access to a device can be performed via:

Port‑I/O instructions (e.g., IN accumulator, {port number | DX} and OUT {port number | DX}, accumulator on x86).

Memory‑mapped I/O, which is generally easier to use.

PCIe Devices on Linux

Linux represents each PCIe device under /sys/bus/pci/devices/ with files such as:

config : binary configuration space (read/write).

device : Device ID (read‑only).

vendor : Vendor ID (read‑only).

driver : linked driver directory.

enable : device enable flag (read/write).

irq : assigned interrupt number.

local_cpulist / local_cpu : CPUs sharing the same NUMA node.

numa_node : NUMA node of the device.

resource, resource0…N : BAR address ranges.

sriov_numfs / sriov_totalvfs : number of virtual functions.

subsystem_device / subsystem_vendor : subsystem IDs.

Viewing a Device’s BDF

An example shows a Starblaze STAR1000 NVMe SSD with BDF 3C:00.0 (Bus 0x3C, Device 0x00, Function 0).

Viewing Vendor and Device IDs

The device’s Vendor ID is assigned by PCI‑SIG, while the Device ID is chosen by the vendor. Class code indicates the device type (e.g., NVMe SSD).

Viewing Detailed Device Information

The detailed view reveals BAR sizes (e.g., a 1 MiB Memory BAR and a 256 KiB BAR), confirming that the OS can access the device through its memory space.