How NVMe over Fabrics Will Change the Storage Environment

Before diving into NVMe over Fabrics, it is useful to recall the evolution of storage networking: Fibre Channel became popular in the late 1990s, and later iSCSI offered an Ethernet‑based alternative for those unwilling to invest in dedicated Fibre Channel hardware. Both rely on SCSI as the protocol between initiators and targets, but the rise of flash storage exposed performance limits of SCSI.

Media Technology Hits a Bottleneck

Traditional storage protocols such as parallel SCSI, SAS, and SATA were designed for spinning disks and provide limited command queue depth (e.g., AHCI’s single queue of 32 commands). Even with serial interfaces, the bandwidth caps (6 Gbps for SATA 3.0, 12 Gbps for SAS 3.0) and single‑queue latency become bottlenecks for modern SSDs, which can handle many concurrent I/O operations.

Introducing NVMe

NVMe (Non‑Volatile Memory Express) moves the storage protocol onto the PCIe bus, offering roughly 4 Gbps per lane and dramatically lower latency. Its biggest advantage is the support for up to 65,535 I/O queues, each with 65,535 commands, allowing massive parallelism that matches today’s multi‑core processors.

The NVM Express Working Group, formed by about 90 companies, released the NVMe 1.3 specification in July 2012, adding features such as security, resource sharing, and SSD durability management.

What Is NVMe over Fabrics?

NVMe can also replace SCSI in networked environments, leading to the NVMe‑over‑Fabrics (NVMe‑of‑F) standards introduced in 2014. Two transport types are being developed: RDMA‑based NVMe‑of‑F and Fibre Channel‑based FC‑NVMe.

RDMA enables direct memory‑to‑memory data transfer without CPU involvement, delivering ultra‑low latency over InfiniBand, iWARP, or RoCE. Vendors such as Mellanox provide 100 Gbps adapters that offload NVMe‑of‑F processing.

Implementing NVMe

In data centers, the most visible deployment is installing NVMe SSDs directly in servers, with many vendors offering NVMe‑ready hardware, BIOS support, and hypervisor integration (e.g., VMware vSphere and vSAN).

Alternatively, NVMe can serve as the back‑end protocol for storage arrays, replacing SAS or Fibre Channel. This shift yields significant performance gains because flash‑based arrays can fully exploit NVMe’s parallelism.

Major vendors have announced NVMe support: HPE’s 3PAR, NetApp’s FlashCache, Pure Storage’s FlashArray//X (claiming half the latency and double the write bandwidth of previous generations), though host‑side NVMe‑of‑F support may still be pending.

Choices and Migration Paths

Adopting NVMe often means moving to an all‑NVMe SAN architecture, with options for FC‑NVMe or Ethernet‑based implementations. Cisco’s MDS 9710 and Brocade’s Gen6 32 Gbps switches already support FC‑NVMe.

Customers can avoid costly Fibre Channel upgrades by leveraging existing 32 Gbps infrastructure, but servers lacking 32 Gbps HBAs may need new hardware.

Because NVMe and SCSI can coexist, many data centers can transition gradually, preserving familiar management tools while introducing NVMe where performance benefits are most needed.

The Future of NVMe

NVMe is poised to become the default connection for SSDs, with high‑end deployments relying on NVMe‑over‑Fabrics to retain features such as snapshots, replication, compression, and deduplication. Emerging platforms like Excelero’s NVMesh and Apeiron’s 40 GbE NVMe arrays illustrate the ongoing innovation that will eventually make NVMe the dominant storage protocol.