Why InfiniBand Is Outpacing Ethernet in High‑Performance Computing

Introduction

With CPU performance soaring, high‑speed interconnects (HSI) have become critical for high‑performance computing (HPC). After years of development, the dominant HSI technologies are Gigabit Ethernet and InfiniBand, the latter growing fastest under the guidance of the InfiniBand Trade Association (IBTA).

InfiniBand Trade Association (IBTA)

Founded in 1999 by merging two industry groups, IBTA is led by a steering committee that includes HP, IBM, Intel, Mellanox, Oracle, QLogic, Dell, Bull and others. It conducts compliance and interoperability testing and drives the evolution of the InfiniBand specifications.

InfiniBand Overview

InfiniBand is a point‑to‑point, switched fabric designed for processor‑to‑I/O communication, supporting up to 64,000 addressable devices. Its architecture (IBA) defines a standard framework used in servers, storage, networking, and embedded systems. Key features include high bandwidth, low latency, low management cost, and scalability to thousands of nodes.

InfiniBand inherits the bus‑level bandwidth and low latency of traditional processor buses, implementing Direct Memory Access as Remote Direct Memory Access (RDMA). RDMA enables zero‑copy data transfer between nodes without OS involvement, reducing CPU overhead and latency.

Advantages of InfiniBand

Standard: IBTA’s 300+ members define open standards supporting SRP and iSER storage protocols.

Speed: Current rates reach 168 Gbps (12×FDR), far exceeding 10 Gbps Fibre Channel and 100 Gbps Ethernet.

Memory: Host Channel Adapters (HCA) provide RDMA, allowing direct memory‑to‑memory transfers without kernel intervention.

Transport Offload: RDMA offloads packet routing to the NIC, freeing CPU cycles.

QoS: Multi‑level quality‑of‑service guarantees diverse service requirements.

InfiniBand Packet Structure

A complete InfiniBand packet consists of several fields:

LRH (Local Route Header): 8 bytes – identifies source/destination ports, service level, and virtual lane.

GRH (Global Route Header): 40 bytes – IPv6‑style routing information for inter‑subnet packets.

BTH (Base Transport Header): 12 bytes – contains queue pair, opcode, sequence number, and segmentation info.

ETH (Extended Transport Header): 4‑28 bytes – provides reliable datagram service.

Payload (PYLD): 0‑4096 bytes – application data.

ICRC (Invariant CRC): 4 bytes – invariant checksum.

VCRC (Variant CRC): 2 bytes – variable checksum for raw packet reconstruction.

InfiniBand uses a 128‑bit IPv6‑style address in the GRH, and its physical layer supports copper and optical cables with 1X, 4X, 8X, and 12X configurations. Supported data rates include SDR, DDR, QDR, FDR, and EDR, making it suitable for large‑scale data transfers such as distributed databases and data‑mining workloads.

InfiniBand Layered Architecture

According to IBTA, InfiniBand consists of four layers:

Physical Layer: Provides signaling, connectors, power management, and encoding for the link layer.

Link Layer: Handles packet framing, addressing, flow control, error detection, and QoS.

Network Layer: Routes packets between subnets, supporting unicast and multicast.

Transport Layer: Adds transport headers, manages queue pairs, and defines reliable or unreliable services via verbs.

Switching Mechanism

InfiniBand employs a switched‑fabric topology. Switches forward packets based on the destination Local Identifier (DLID), select the appropriate virtual lane (VL) using SL‑VL mapping, perform credit‑based flow control, and support unicast, multicast, and broadcast. They also enforce partitioning via partition keys and perform extensive error checking.

Comparison with Ethernet

Benchmark tables (omitted) show InfiniBand’s superior bandwidth and latency compared to Ethernet, giving it a clear advantage in HPC workloads. Market data from the latest TOP500 list indicate InfiniBand’s share in the top 100 supercomputers is rising, while Ethernet’s share is declining.

Conclusion and Outlook

InfiniBand is poised to replace 10 GbE/40 GbE as the preferred high‑speed interconnect. IBTA forecasts rapid growth for FDR, EDR, and HDR over the next three years, with bandwidth potentially reaching 1 Tbps before 2020. Future applications include GPU clusters, NVMe‑based storage, and large‑scale database systems.