Understanding SR‑IOV: Concepts, Benefits, and AWS Nitro Implementation in Cloud Data Centers

In September 2007 the PCI‑SIG released the "Single Root I/O Virtualization and Sharing Specification Revision 1.0," defining how multiple system images can share PCIe I/O devices such as network cards or SSDs.

SR‑IOV is a hardware‑oriented virtualization solution that splits a Physical Function (PF) into multiple Virtual Functions (VFs) which can be directly assigned to virtual machines, allowing them to bypass the Virtual Intermediary (VI) or hypervisor and achieve near‑bare‑metal I/O performance.

Key concepts include:

System Image (SI) – the guest OS or virtual machine.

Virtual Intermediary (VI) – the hypervisor or VMM that normally mediates I/O.

SR‑PCIM – the software that configures and manages SR‑IOV, handling errors, power management, and hot‑plug of VFs.

Physical Function (PF) – a PCIe physical function that can be discovered and managed by the host.

Virtual Function (VF) – a lightweight virtual function created from a PF, presented to a VM as a regular PCIe device.

Benefits of SR‑IOV include eliminating hypervisor I/O overhead, reducing CPU load on the host, enabling multiple VMs to share high‑performance devices such as NVMe SSDs, decreasing the number of required PCIe slots, and allowing combination with other I/O virtualization techniques for a secure, high‑performance solution.

In cloud data centers, AWS has been a leader in applying SR‑IOV through its Nitro system. After acquiring Annapurna Labs in 2015, AWS introduced Nitro‑based instances (e.g., C5, C5D) where storage and network traffic are delivered via SR‑IOV, achieving performance close to bare metal with only about 10 µs additional latency.

Performance measurements on an AWS c5d.large instance show NVMe instance‑store read latency around 96 µs and write latency between 24 µs and 37 µs. Both EBS volumes and local NVMe SSDs appear to the VM as standard NVMe block devices (e.g., /dev/nvme0n1, /dev/nvme1n1).

Memblaze’s testing confirms these results and highlights further requirements for SSDs to support SR‑IOV, such as multi‑namespace management for tenant isolation, driver modifications for PCIe BAR allocation and NVMe I/O timeout handling, and extensive co‑validation with customers to ensure reliability and performance.

Overall, SR‑IOV provides a high‑performance, secure, and scalable I/O virtualization method that is especially valuable in cloud and virtualized environments, as demonstrated by AWS’s Nitro implementation and ongoing industry research.