NVMe over RoCEv2 Network Architecture, Control Optimization Requirements, and Test Specification

Historically, high‑performance storage applications relied on Fibre Channel (FC) networks; with faster transport speeds, media have evolved from HDDs to SSDs and protocols from SCSI to NVMe. NVMe‑over‑RoCEv2, the most NVMe‑semantic‑preserving implementation of NVMe‑over‑Fabric, surpasses FC in performance, cost, and manageability, positioning it as the future of high‑speed storage networking.

The NVMe‑over‑RoCEv2 specification, defined by the NVM Express organization, optimizes network control for usability, maintainability, and reliability, making it suitable for mission‑critical workloads that demand high reliability.

NVMe over RoCEv2 Topology

The network consists of three roles: initiators (hosts), switches, and targets (storage). Hosts and storage are endpoint devices that exchange data via the NVMe‑over‑RoCEv2 protocol to provide high‑performance storage services.

Network control optimization requires coordinated plug‑and‑play and rapid fault detection across hosts, switches, and storage. Plug‑and‑play mandates automatic device discovery by switches, synchronization across the fabric, and notification to subscribed hosts, which then establish connections to storage. This functionality supports initial deployment, scaling, and maintenance.

Rapid fault detection requires switches to detect failures, propagate the status, and notify subscribed hosts. Hosts must determine whether the affected device is storage, promptly disconnect, and trigger multipath software to switch to a redundant path.

Network Control Optimization Technical Requirements

1. Business Functions and Processes

Switches act as the network core, managing IP domain information, device registration, and status monitoring. They must synchronize IP domain and device state across all switches and notify subscribed nodes of changes.

Hosts, as storage service consumers, send registration messages, periodically announce presence, subscribe to network status updates, and react to device join/leave events by establishing or tearing down NVMe‑oF connections.

Storage, as the service provider, registers itself, optionally subscribes to network events for location awareness, and participates in plug‑and‑play and fault‑detection workflows.

2. IP Domain Management

Switches implement IP domain management, allowing administrators to configure, add, delete, or modify domains, support import/batch configuration, default domains, IP aliases, and address ranges, and ensure consistent domain information across the fabric.

3. LLDP Announcement

Endpoints (hosts, storage, switches) use LLDP extended TLVs to announce chassis ID (MAC address) and port ID (prefix "snsd_" + IP‑based port name). Switches send LLDP every 30 seconds, with a 120‑second aging timer.

LLDP technical requirements for hosts and storage include periodic transmission, per‑IP transmission, updates on port changes, suspension during network faults, and cessation upon IP or VLAN removal. Aggregated ports must announce each member.

Switches must receive and parse LLDP, synchronize device information across the fabric, notify IP domain devices of new endpoints, update information on changes, delete stale entries after aging, and support at least 64 neighbors per port.

4. State Notification

State‑notification messages consist of TLVs describing online/offline events, generated only by access switches and sent to subscribed endpoints, which must acknowledge receipt. Messages are sent in network byte order.

Switches must send notifications only to subscribed devices, detect network faults or configuration changes, synchronize state across the fabric, retry up to three times (intervals 100 ms, 1 s, 10 s), and deliver notifications within 500 ms of a fault.

Hosts must subscribe to state notifications, acknowledge them, deduplicate duplicate messages, establish NVMe‑oF connections on storage join, and disconnect within 500 ms on storage leave.

Storage must optionally subscribe, acknowledge, and deduplicate notifications similarly.

5. Information Synchronization

Switches synchronize IP domain configuration and device status across the fabric, ensuring timeliness and consistency.

6. Network Security Requirements

Switches must perform validity checks, protect against DDoS and LLDP spoofing, prevent tampering of synchronization data, and log or alarm on anomalies. Hosts and storage must also perform validity checks, DDoS protection, and reject spoofed network notifications.

Network Control Optimization Test Specification

1. Test Scenario Analysis

Two major scenarios are evaluated: plug‑and‑play and rapid‑fault‑perception, with additional security tests for abnormal packet attacks.

2. Test Tools

Four test scenarios cover device insertion, link failure, fabric partition, and IP domain updates, verifying that hosts automatically detect storage loss, generate appropriate notifications, and trigger multipath software to switch paths.

Download link: NVMe over RoCEv2 Network Optimization Requirements and Test Specification

Related links: Cloud Computing Research Report , RDMA Technology Whitepaper , NVMe Storage Acceleration with SPDK