Understanding NVMe over Fabrics: Architecture, Benefits, and RDMA Integration

On June 9, 2016, NVM Express announced the NVMe over Fabrics protocol, marking a pivotal shift for storage vendors as they seek to extend NVMe’s low‑latency, high‑throughput benefits beyond a single host.

Modern SSDs built on flash or PCM dramatically change enterprise storage, and NVMe, designed for non‑volatile memory, optimizes performance, latency, and I/O stack overhead, making it ideal for today's demanding applications.

To fully exploit these SSD capabilities, storage capacity must be pooled and shared across compute nodes, aligning with virtualization, software‑defined storage, and cloud‑native models that deliver cost savings, reliability, and easier management.

Market analysis shows traditional DAS, SAN, and NAS are being supplanted by large‑scale enterprise cloud and Server‑SAN solutions, driven by the feasibility and adoption speed of new technologies like NVMe over Fabrics.

NVMe over Fabrics extends NVMe’s high performance, low latency, and minimal protocol overhead across networked storage, allowing hosts to access any data‑center node with the same efficiency as local NVMe devices.

The protocol is built atop existing transport layers—RDMA, Fibre Channel, PCIe Fabrics—so it can leverage current hardware without discarding established data‑center infrastructure.

Prior storage interconnects such as iSCSI (IP‑based) and its RDMA‑enhanced variant iSER improved performance but remained limited by the SCSI stack; NVMe over Fabrics was created to overcome these constraints.

NVMe over Fabrics defines bindings for various transports, including standards like FC‑NVMe, and details how each transport’s characteristics affect implementation complexity and hardware requirements.

Transport interfaces are categorized into memory‑type, message‑type, and hybrid interfaces, illustrated with diagrams in the original article.

RDMA, an older technology, enables direct memory transfers over the network with minimal CPU involvement, reducing data copies and latency, which is crucial for high‑performance NVMe traffic.

RDMA operations use verbs split into control‑path (resource management) and data‑path (send/receive) primitives, and rely on memory regions (MR) to describe remote memory, allowing zero‑copy data movement.

By mapping NVMe commands onto RDMA’s low‑latency, zero‑copy mechanisms, NVMe over Fabrics achieves efficient network‑based storage, though differences from PCIe require careful protocol adaptation.

The article concludes that NVMe over Fabrics builds on the NVMe Base specification (revision 1.2.1) and that future posts will delve into the specific challenges and I/O transmission processes the protocol addresses.