How PCIe Configuration Impacts Network Performance: A Practical Guide

PCIe Uses

When PCIe connects a network adapter to a host, the adapter must communicate with the CPU, memory, and other modules, so proper PCIe configuration is essential for handling network traffic efficiently. The connection automatically negotiates the maximum capabilities supported by both the adapter and the CPU.

PCIe Attributes

All PCI devices expose attributes that are negotiated between the system and the device, determining the highest values both can support. The following sections describe key PCIe attributes, how to verify them, and their impact on performance.

PCIe Width

PCIe width defines the number of parallel lanes available for communication, expressed as xA (e.g., x8 for eight lanes). The supported lane count depends on the device model. Verify the width with the lspci command.

# lspci -s 04:00.0 -vvv | grep Width
LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM not supported, Exit Latency L0s unlimited, L1 unlimited
LnkSta: Speed 8GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-

PCIe Speed

Speed, measured in GT/s (giga‑transactions per second), together with width determines the maximum PCIe bandwidth (speed × width). Verify speed with lspci.

# lspci -s 04:00.0 -vvv | grep Speed
LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM not supported, Exit Latency L0s unlimited, L1 unlimited
LnkSta: Speed 8GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt

PCIe generations map to speeds: gen1 = 2.5 GT/s, gen2 = 5 GT/s, gen3 = 8 GT/s, gen4 = 16 GT/s. Higher generations reduce header overhead, improving effective bandwidth.

PCIe Max Payload Size

Max Payload Size defines the largest PCIe packet (or MTU) that can be transmitted. It is set by the chipset and can be inspected via lspci under the DevCtl field.

lspci -s 04:00.0 -vvv | grep DevCtl: -C 2
DevCap: MaxPayload 512 bytes, ...
DevCtl: ...
MaxPayload 256 bytes, MaxReadReq 4096 bytes

PCIe Max Read Request

Max Read Request specifies the maximum size of a read request, influencing how many pending requests a device can handle. Verify and modify it with lspci and setpci.

# lspci -s 04:00.0 -vvv | grep MaxReadReq
MaxPayload 256 bytes, MaxReadReq 4096 bytes

# setpci -s 04:00.0 68.w
5936

# setpci -s 04:00.0 68.w=2936

# lspci -s 04:00.0 -vvv | grep MaxReadReq
MaxPayload 256 bytes, MaxReadReq 512 bytes

Acceptable values are 0‑128 B, 1‑256 B, 2‑512 B, 3‑1024 B, 4‑2048 B, and 5‑4096 B.

Calculating PCIe Bandwidth Limits

The theoretical maximum PCIe bandwidth is obtained by multiplying speed and width, then subtracting roughly 1 Gb/s for error‑correction and header overhead. The formula is:

Maximum PCIe bandwidth = speed × width × (1 – encoding) – 1 Gb/s.

Example 1: A gen3 (8 GT/s) device with x8 width:

Maximum bandwidth ≈ 8 G × 8 × (1 – 2/130) – 1 G ≈ 62 Gb/s.

Example 2: A gen2 (5 GT/s) device with x16 width:

Maximum bandwidth ≈ 5 G × 16 × (1 – 1/5) – 1 G ≈ 63 Gb/s.

Note that PCIe transactions include both payload and header, so both must be considered when evaluating network traffic limits. Additionally, Max Read Request and Max Payload Size can affect transaction rates by influencing how many pending requests can be processed.