How NVMe over Fabrics Is Transforming Modern Storage Networks

Background: From SCSI to NVMe

Storage networking began with Fibre Channel in the late 1990s, followed by iSCSI as an Ethernet‑based alternative. Both rely on the SCSI protocol, which became a performance bottleneck as flash storage replaced spinning disks. Parallel SCSI, SAS, and SATA all suffer limited queue depth and bandwidth, leading to high latency for SSDs.

Why NVMe Was Needed

NVMe (Non‑Volatile Memory Express) was created to overcome SCSI’s limitations by using the PCIe bus, offering much higher bandwidth (e.g., PCIe 3.0 × 4 ≈ 4 GB/s) and lower latency. It supports up to 65,535 I/O queues, each with 65,535 commands, enabling massive parallelism that matches modern multi‑core CPUs.

NVMe over Fabrics (NVMe‑of)

NVMe‑of extends the NVMe protocol across network fabrics, allowing NVMe to replace SCSI not only on local buses but also over RDMA‑based fabrics and Fibre Channel (FC‑NVMe). Two main transport types are under development:

RDMA‑based NVMe‑of, using technologies such as InfiniBand, iWARP, or RoCE.

FC‑NVMe, which runs NVMe over traditional Fibre Channel.

RDMA enables direct memory transfers between hosts without CPU involvement, delivering sub‑microsecond latency. Vendors like Mellanox provide 100 Gbps adapters that offload NVMe‑of processing.

Implementation Options

Enterprises can adopt NVMe in two primary ways:

Deploy NVMe SSDs directly in servers; most modern servers and hypervisors (e.g., VMware vSphere, vSAN) already support them.

Use NVMe as the back‑end protocol for storage arrays, replacing SAS or Fibre Channel. This approach leverages existing SAN infrastructure while gaining the performance benefits of NVMe.

Major vendors have announced NVMe support: HPE’s 3PAR, NetApp’s FlashCache, Pure Storage’s FlashArray//X (claiming half the latency and double the write bandwidth of previous generations), Cisco’s MDS 9710 Director with FC‑NVMe, and Brocade’s Gen6 32 Gbps switches.

Vendor‑Specific Deployments

Start‑ups are also pushing the envelope:

E8 Storage offers a 100 GbE‑based NVMe‑of SAN claiming up to 10 M reads and 2 M writes per second with 100 µs read latency.

Excelero’s NVMesh creates a distributed NVMe fabric using Mellanox RoCE adapters and Micron 3.2 TB SSDs.

Apeiron Data Systems is developing a 40 GbE NVMe array architecture for hyper‑converged designs.

Future Outlook

NVMe is poised to become the default protocol for SSDs, eventually supplanting SCSI and SAS. High‑end deployments will rely on NVMe‑of to retain features such as snapshots, replication, compression, and deduplication while delivering dramatically lower latency and higher throughput. As more storage arrays and data‑center fabrics adopt NVMe, the industry will shift away from legacy Fibre Channel and Ethernet‑only SANs toward unified, high‑performance NVMe fabrics.