How to Build High‑Performance RDMA Applications in Go with rsocket

RDMA Background

Remote Direct Memory Access (RDMA) bypasses the kernel to read or write remote memory directly, saving CPU cycles and dramatically reducing latency, which makes it ideal for large‑scale parallel clusters, NVMe‑over‑Fabric storage networks, and AI model training workloads.

Common RDMA transports include RoCE, InfiniBand, and iWARP, all requiring compatible NIC hardware.

libibverbs API and rdma_cm

Traditional RDMA programming uses the libibverbs API, which exposes many low‑level concepts (queue pairs, completion queues, memory registration) and requires a lengthy initialization sequence even for a simple "hello‑world" program.

To simplify connection management, the RDMA Communication Manager (rdma_cm) abstracts address resolution and connection setup, supporting multiple transports and providing a unified, event‑driven interface.

rsocket: A Socket‑Like RDMA Interface

rsocket, designed by Mellanox (now NVIDIA) and standardized by the OpenFabrics Alliance, offers a BSD‑style socket API that internally maps calls to RDMA verbs, allowing developers to write familiar socket code while gaining RDMA performance.

Key rsocket functions mirror standard socket calls, prefixed with r (e.g., rsocket, rbind, rlisten, raccept, rconnect, rread, rwrite, rpoll, rselect), and the API also defines options such as SOL_RDMA and enumerations for queue sizes and inline thresholds.

Why Not Use VMA Directly?

Mellanox’s VMA library can intercept standard BSD socket calls via LD_PRELOAD=libvma.so, providing kernel‑bypass acceleration for InfiniBand and RoCE NICs. However, VMA is tightly coupled to Mellanox hardware and offers limited support for other vendors, making a portable, vendor‑agnostic solution like rsocket more attractive for cross‑platform development.

Go Wrapper: github.com/smallnest/rsocket

The smallnest/rsocket repository wraps the rsocket C API with CGO, exposing Go functions that correspond to the rsocket primitives (e.g., Socket, Bind, Listen, Accept, Connect, Read, Write) and also implements the net.Conn interface for idiomatic Go usage.

TCP Server Example

package main

import (
    "fmt"
    "log"
    "net"
    "syscall"
    "github.com/smallnest/rsocket"
)

func main() {
    // Create RDMA socket
    fd, err := rsocket.Socket(rsocket.AF_INET, rsocket.SOCK_STREAM, 0)
    if err != nil { log.Fatal(err) }
    defer rsocket.Close(fd)

    // Bind to address
    addr := &net.TCPAddr{IP: net.ParseIP("0.0.0.0"), Port: 8000}
    sa := &syscall.SockaddrInet4{Port: addr.Port}
    copy(sa.Addr[:], addr.IP.To4())
    if err := rsocket.Bind(fd, sa); err != nil { log.Fatal(err) }

    // Listen for connections
    if err := rsocket.Listen(fd, 128); err != nil { log.Fatal(err) }
    fmt.Printf("Server listening on :%d
", addr.Port)

    for {
        clientFd, clientAddr, err := rsocket.Accept(fd)
        if err != nil { log.Printf("Accept error: %v", err); continue }
        go handleConnection(clientFd, clientAddr)
    }
}

func handleConnection(clientFd int, clientAddr syscall.Sockaddr) {
    defer rsocket.Close(clientFd)
    if addr, ok := clientAddr.(*syscall.SockaddrInet4); ok {
        ip := net.IPv4(addr.Addr[0], addr.Addr[1], addr.Addr[2], addr.Addr[3])
        fmt.Printf("New client: %s:%d
", ip.String(), addr.Port)
    }
    buffer := make([]byte, 1024)
    n, err := rsocket.Read(clientFd, buffer)
    if err != nil { log.Printf("Read error: %v", err); return }
    fmt.Printf("Received: %s
", buffer[:n])
    response := []byte("Server received your message!")
    n, err = rsocket.Write(clientFd, response)
    if err != nil { log.Printf("Write error: %v", err); return }
    fmt.Printf("Sent %d bytes response
", n)
}

TCP Client Example

package main

import (
    "fmt"
    "log"
    "net"
    "syscall"
    "github.com/smallnest/rsocket"
)

func main() {
    // Create RDMA socket
    fd, err := rsocket.Socket(rsocket.AF_INET, rsocket.SOCK_STREAM, 0)
    if err != nil { log.Fatal(err) }
    defer rsocket.Close(fd)

    // Server address
    serverAddr := &net.TCPAddr{IP: net.ParseIP("127.0.0.1"), Port: 8000}
    sa := &syscall.SockaddrInet4{Port: serverAddr.Port}
    copy(sa.Addr[:], serverAddr.IP.To4())

    // Connect to server
    if err := rsocket.Connect(fd, sa); err != nil { log.Fatal("Connect failed:", err) }
    fmt.Println("Connected to server")

    // Send data
    msg := []byte("Hello, RDMA Server!")
    n, err := rsocket.Write(fd, msg)
    if err != nil { log.Fatal("Write failed:", err) }
    fmt.Printf("Sent %d bytes
", n)

    // Receive response
    buf := make([]byte, 1024)
    n, err = rsocket.Read(fd, buf)
    if err != nil { log.Fatal("Read failed:", err) }
    fmt.Printf("Server reply: %s
", buf[:n])
}

The library also provides a net.Conn implementation, allowing Go developers to use the standard net package patterns while the underlying transport remains RDMA‑accelerated.

References

RDMA Column – https://zhuanlan.zhihu.com/p/164908617

What is RDMA? – https://wiki.h3c.com/cn/detail.html?WikiName=RDMA

libibverbs API – https://github.com/linux-rdma/rdma-core/tree/master/libibverbs

rsocket man page – https://linux.die.net/man/7/rsocket

rsocket header – https://github.com/ofiwg/librdmacm/blob/master/include/rdma/rsocket.h

smallnest/rsocket repository – https://github.com/smallnest/rsocket