How PCIe Configuration Impacts Performance

PCIe Usage

When PCIe is used for communication between a network adapter and the host, the adapter must talk to the CPU, memory and other modules. Proper configuration of the devices that use PCIe is required to handle network traffic efficiently. The adapter automatically negotiates the maximum capabilities supported by both the adapter and the CPU.

PCIe Attributes

All PCI devices expose a set of attributes, some of which are critical for performance. These attributes are negotiated between the system and the device, allowing both sides to support the highest common values. The article lists the key attributes, how to verify them, and their performance impact.

PCIe Width

PCIe width determines how many parallel lanes are available for communication, expressed as xA (e.g., x8). The supported lane count depends on the vendor and model. The width can be verified with # lspci -s 04:00.0 -vvv | grep Width, which on a Mellanox adapter at 04:00.0 reports:

LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM not supported, Exit Latency L0s unlimited, L1 unlimited
LnkSta: Speed 8GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-

The LnkCap line shows the device’s capabilities; LnkSta shows the current negotiated state.

PCIe Speed

Speed, measured in GT/s, together with width determines the maximum PCIe bandwidth (speed × width). Verify speed with # lspci -s 04:00.0 -vvv | grep Speed which also reports the same 8GT/s and x8 values. PCIe generations map to speeds: gen1 = 2.5GT/s, gen2 = 5GT/s, gen3 = 8GT/s, gen4 = 16GT/s. The main difference between generations is packet encoding overhead (20 % for gen1/2, 1.5 % for gen3).

PCIe Max Payload Size

Max Payload Size defines the largest PCIe packet (or MTU) that can be transmitted. Larger payloads reduce the number of packets needed for a transaction. The value is set by the system based on the chipset architecture and can be inspected with # lspci -s 04:00.0 -vvv | grep DevCtl -C 2, which shows:

DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 256 bytes, MaxReadReq 4096 bytes

PCIe Max Read Request

Max Read Request specifies the largest read request the device may issue. It influences how many outstanding read requests can be buffered. Query with # lspci -s 04:00.0 -vvv | grep MaxReadReq to see “MaxReadReq 4096 bytes”. The value can be changed at runtime with # setpci -s 04:00.0 68.w=2936 after first reading the current value with # setpci -s 04:00.0 68.w. Valid values are 0‑5 corresponding to 128 B, 256 B, 512 B, 1 KB, 2 KB, 4 KB.

# setpci -s 04:00.0 68.w
5936
# setpci -s 04:00.0 68.w=2936
# lspci -s 04:00.0 -vvv | grep MaxReadReq
MaxPayload 256 bytes, MaxReadReq 512 bytes

Calculating PCIe Bandwidth Limits

The theoretical maximum bandwidth is speed × width, reduced by encoding overhead and about 1 Gb/s for error‑correction and header overhead. Formula:

Max PCIe bandwidth = speed × width × (1 − encoding) − 1 Gb/s.

Example 1: gen3 (8 GT/s) with x8 width → 8 G × 8 × (1 − 2/130) − 1 ≈ 62 Gb/s.

Example 2: gen2 (5 GT/s) with x16 width → 5 G × 16 × (1 − 1/5) − 1 ≈ 63 Gb/s.

When evaluating network traffic, remember that PCIe transactions include both payload and header, and that Max Read Request and Max Payload Size can further limit transaction rates.