Why InfiniBand Is Outpacing Ethernet in High‑Performance Computing

1. Introduction

InfiniBand has emerged as one of the fastest-growing high‑speed interconnect technologies, offering high bandwidth, low latency, and easy scalability, making it a key component in modern high‑performance computing (HPC) systems.

2. InfiniBand Trade Association (IBTA)

Founded in 1999 by merging two industry groups, IBTA oversees the development and compliance testing of InfiniBand specifications. Its members—including major vendors such as Mellanox, Intel, IBM, and Dell—collaborate to define and update the open standards.

3. Overview of InfiniBand

InfiniBand provides a point‑to‑point, switched‑fabric communication link between processors and I/O devices, supporting up to 64,000 addressable devices. The architecture defines a subnet as the smallest complete unit, with subnets interconnected by routers to form larger networks.

Key features include:

High bandwidth and low latency derived from bus‑like characteristics.

Remote Direct Memory Access (RDMA) enables zero‑copy data transfer without OS intervention.

Support for multiple data streams (cluster, storage, management) over a single connection.

4. InfiniBand Architecture Layers

The InfiniBand architecture consists of four layers:

Physical Layer: Provides signaling, connectors, power management, and encoding.

Link Layer: Handles data framing, flow control, error detection, and QoS.

Network Layer: Routes packets between subnets, supporting unicast and multicast.

Transport Layer: Adds transport headers, manages queue pairs (QP), and defines reliable and unreliable services.

5. Packet Structure

An InfiniBand packet comprises several fields:

Local Route Header (LRH): 8 bytes, identifies source/destination ports and virtual lane.

Global Route Header (GRH): 40 bytes, provides IPv6‑style routing across subnets.

Base Transport Header (BTH): 12 bytes, specifies queue pair, opcode, and sequence number.

Extended Transport Header (ETH): 4‑28 bytes, offers additional transport services.

Payload (PYLD): 0‑4096 bytes of user data.

Invariant CRC (ICRC): 4 bytes error‑checking.

Variant CRC (VCRC): 2 bytes variable error‑checking.

InfiniBand uses a 128‑bit IPv6‑compatible address scheme, enabling direct routing between host channel adapters (HCA) and target channel adapters (TCA).

6. Performance Advantages

Key advantages include:

Standardization: Open specifications support SRP and iSER storage protocols.

Speed: Current implementations reach 168 Gbps (12×FDR), far exceeding 10 Gbps Ethernet and 100 Gbps Ethernet.

Memory Access: RDMA offloads data movement to hardware, eliminating OS copies.

Transport Offload: Packet processing occurs at the NIC, reducing CPU load.

QoS: Multi‑level quality‑of‑service guarantees.

7. Comparison with Ethernet

Benchmark tables show InfiniBand delivering significantly higher throughput and lower latency than both 10 Gbps Ethernet and 100 Gbps Ethernet. Its cost per performance is also competitive, and market surveys (e.g., TOP500 HPC rankings) indicate a growing share of InfiniBand over Ethernet.

8. Future Outlook

IBTA forecasts rapid growth for HDR, EDR, and FDR variants, with bandwidth expectations reaching 1 Tbps before 2020. Anticipated applications include GPU acceleration, NVMe‑over‑Fabric, and large‑scale database clusters.