How GPUDirect P2P Boosts Multi‑GPU Performance and What Limits It in Virtualized Environments

Background

GPUs are crucial for high‑performance computing and deep‑learning acceleration due to their massive parallelism, but as data volumes grow, inter‑GPU communication becomes a bottleneck, making communication performance a key metric.

GPUDirect Overview

NVIDIA introduced GPUDirect to improve GPU communication, but PCI‑Express limitations restrict bandwidth, leading to the development of the NVLink protocol.

Key Features

Accelerated communication with network and storage devices

GPU‑to‑GPU Peer‑to‑Peer transfers

Peer‑to‑Peer memory access

RDMA support

Video‑specific optimizations

Technology Variants

Shared Memory : Introduced in June 2010, enables GPUs and third‑party PCI‑Express devices to share pinned host memory for faster data exchange.

P2P : Added in 2011, supports direct access and transfers between GPUs under the same PCI‑Express root complex.

RDMA : Added in 2013, allows third‑party PCI‑Express devices to bypass CPU host memory and access GPU memory directly.

GPUDirect Peer‑to‑Peer (P2P)

GPUDirect P2P enables single‑node GPUs to communicate over PCI‑Express without copying data to CPU host memory, dramatically reducing latency. Major deep‑learning frameworks such as TensorFlow and MXNet, as well as NVIDIA's NCCL library, provide support and optimizations for this technology, yielding near‑linear training speedup on multi‑GPU systems.

Virtualization Challenges

In cloud environments, GPUDirect must be virtualized. PCI pass‑through can grant a VM full control of a GPU, but P2P communication between GPUs in the same VM is often disabled because Intel IOH topology does not support PCI‑e P2P across QPI bridges. Hypervisors present a flattened PCI topology, preventing drivers from detecting true hardware layout.

To enable P2P in virtualized settings, a PCI capability indicating GPU P2P affinity must be added to the emulated PCI configuration space, allowing the driver to activate P2P. Additionally, all PCI‑Express traffic, including P2P, is routed through the IOMMU in pass‑through scenarios, introducing slight extra latency compared to bare metal.

Experimental Results

Measurements were performed on an Alibaba Cloud GN5 instance equipped with eight Tesla P100 GPUs.

GPU‑to‑GPU latency matrix and comparison charts show that enabling GPUDirect P2P cuts communication latency by nearly 50% relative to CPU‑mediated copies.

Training a classic convolutional neural network with MXNet on this instance demonstrates excellent single‑node scaling, achieving almost linear speedup when P2P is enabled.