The article analyzes China's first ten-thousand-card intelligent computing cluster network technology report, detailing how Sugon's scaleFabric domestic InfiniBand native RDMA system overcomes scaling bottlenecks through hardware-native lossless RDMA, hierarchical supernode/supercluster architecture, large subnet expansion to 114,623 cards, and high-reliability multi-tenant operations, validated on the 100K-card 曙光8000 super-intelligent fusion cluster.
Apple's unreleased Harbor app uses Thunderbolt 5 RDMA and MLX's jaccl to link multiple Macs into a private AI inference cluster with a zero-command GUI, but lacks official launch, benchmarks, and may not replace Ollama or vLLM for all workloads.
Apple's Claris subsidiary is testing Harbor, an app that turns Apple Silicon Macs into private AI inference servers with zero-command-line clustering over Thunderbolt 5 to pool unified memory for larger models, though it remains unreleased, unpriced, and enterprise-focused.
Alibaba Cloud presented nine technical sessions at OCP 2026 covering AI network architecture, DCI, optical transport, SONiC, switch chip evaluation, and automated testing, detailing three scaling directions — Scale-up, Scale-out, Scale-across — and open-source engineering practices like Bazel, formal verification, and vSONiC digital twin testing.
DeepSeek's technical report reveals their DeepInfer framework processes 155 trillion tokens daily on just 2,400 GPUs backed by a 100 PiB SSD cache cluster, achieving $38 per GPU per hour by offloading 96% of prefill compute to storage through KVCache and SWA Bounded Replay.
Tencent Cloud details seven papers accepted at SIGCOMM and NSDI that tackle cloud networking challenges for massive AI compute clusters, covering disaggregated DPU architecture, bare-metal AI cloud networking, full RDMA virtualization offload, accelerated flow setup, scalable session tables on commodity DDR, microscopic tracing for heterogeneous gateways, and sub-second failure rerouting.
Baidu's PFS L3 rearchitects parallel file storage for AI production loads with a hybrid data engine (Aries + BlockServer), scalable metadata foundation (MetaDB), high-performance client (RapidFC), RDMA network tuning, transparent tiering to object storage, and AgenticFS for multi-tenant Agent workloads, proven in RL training saving 30k GPU-hours weekly.
The article presents a detailed, step‑by‑step analysis of the L0 resource layer that underpins AI infrastructure, covering GPU scheduling, multi‑tier storage, low‑latency networking, core architectural components, key technologies such as MIG, Volcano, Kueue and RDMA, practical implementation patterns, quantitative acceptance criteria, and common pitfalls with best‑practice mitigations.
The article recounts a near‑automatic library‑development experiment that started from a PRD, split into 15 issues, executed by a Loop Engineering agent, and resulted in a fully functional Go RDMA library while exposing hidden costs, bugs, and lessons learned.
This article explains the fundamentals of Remote Direct Memory Access (RDMA), its low‑latency, zero‑copy and kernel‑bypass mechanisms, programming interfaces, and real‑world applications in data‑center networks, high‑performance computing, and distributed storage, providing developers with practical guidance and code examples.
This article explains the fundamentals of zero‑copy, DMA, PageCache and RDMA, compares them with traditional I/O, describes Linux implementations such as sendfile, mmap+write, splice and Java NIO APIs, and shows practical use‑cases that dramatically reduce CPU load and latency in high‑throughput networking and file handling.
veRoCE is a ByteDance‑engineered RDMA protocol that retains RoCEv2 compatibility while adding multi‑path transmission, native out‑of‑order delivery, efficient loss detection and selective retransmission, allowing high‑performance clusters to operate over lossy Ethernet without PFC.
This article explains the fundamentals of Remote Direct Memory Access (RDMA), its low‑latency, zero‑copy architecture, core principles, programming interfaces, and how it transforms data‑center networking, high‑performance computing, and distributed storage by bypassing the CPU and kernel.
This article explains how traditional kernel network stacks struggle with high‑concurrency, low‑latency cloud data‑center workloads and introduces Baidu Intelligent Cloud’s Polar solution—Polar‑TCP and Polar‑RDMA—which combine user‑space DPDK drivers, a lightweight TCP stack, and an industrial RPC framework to achieve near‑RDMA performance while preserving compatibility with existing TCP ecosystems.
This article explains how traditional kernel network stacks struggle with modern cloud data‑center workloads and introduces Baidu Intelligent Cloud's Polar solution—Polar‑TCP and Polar‑RDMA—which combine user‑space DPDK drivers, a lightweight TCP stack, and an industrial‑grade RPC framework to achieve near‑RDMA performance while preserving ecosystem compatibility.
This guide explains the necessity of flow control in high‑performance storage networks, details RDMA and RoCE protocols, and provides step‑by‑step commands for configuring DSCP, PFC, ECN, and related settings on servers and switches.
This article explains how InfiniBand’s architecture, native RDMA, GPUDirect, and evolving bandwidth enable ultra‑low‑latency, high‑throughput communication for AI model training, compares it with Ethernet, and details the role of RoCEv2 and other high‑performance interconnect technologies.
The article uses a whimsical journey to illustrate how token data is dispatched across GPU clusters—detailing functions like get_dispatch_layout, notify_dispatch, and combine_token, showing RDMA and NVLink pathways, performance experiments, and the final verification of token integrity.
The article explains RDMA's low‑latency, CPU‑bypass communication by detailing the roles of Queue Pairs, Work Queue Entries, and the Doorbell mechanism, comparing them with traditional TCP/IP and outlining their state machines, data structures, and notification workflow.
This article explains the logical and physical architecture of AI compute centers, detailing the division into access, security, network, management, out‑of‑band, AI compute cluster, and general compute zones, and describes the four network planes—parameter, sample, business, and management—required for high‑performance AI workloads.
The article explains how AI‑driven data center networks must handle massive distributed workloads, why traditional Ethernet falls short, and how NVIDIA’s Spectrum‑X Ethernet and Quantum InfiniBand use loss‑less RDMA, dynamic routing, advanced congestion control, and hardware‑accelerated collective communication to deliver the bandwidth, latency, and scalability required for generative AI and large‑scale model training.
This article explains the DCQCN end‑to‑end congestion control algorithm for RoCEv2, describes its three roles (RP, CP, NP), and provides detailed NS‑3 code implementations for ECN marking, CNP generation, and rate‑adjustment logic, including scheduling and parameter calculations.
The article details Huawei’s CloudMatrix384 AI supernode, describing its 384 Ascend 910C NPUs, 192 Kunpeng CPUs, ultra‑high‑bandwidth UB network, three complementary network planes (UB, RDMA, VPC), and the non‑blocking topology that enables sub‑microsecond inter‑node latency across a 16‑rack deployment.
This article explains the performance bottlenecks of traditional I/O, introduces zero‑copy concepts and their relationship with DMA and PageCache, details RDMA architectures, and demonstrates practical zero‑copy implementations such as mmap+write, sendfile, splice, tee, and Java NIO APIs.
This article explains the principles behind zero‑copy, DMA, and RDMA, compares traditional I/O copying with modern zero‑copy techniques, and shows practical implementations in Linux and Java that dramatically reduce CPU overhead and boost network and file‑transfer throughput.
This article details the design and evolution of DHPS, Kuaishou's load‑balanced, RDMA‑based high‑performance service architecture, explaining its network, storage, and traffic‑scheduling innovations that deliver over 270% query‑throughput improvement, lower latency, reduced CPU usage, and near‑five‑nine availability for large‑scale AI inference workloads.
This article explains how data‑center switches are classified, the key components and performance metrics of Ethernet switch chips, market growth trends, the shift from OEO to full‑optical OCS designs, and how RDMA technologies like InfiniBand and RoCEv2 enable the low‑latency networking essential for large‑scale AI training.
3FS, DeepSeek’s high‑performance parallel file system, is engineered for AI workloads, offering ultra‑low latency, high‑throughput storage via RDMA, CRAQ consistency, and seamless cloud‑native integration, with detailed architecture, deployment steps, performance benchmarks, and cost‑saving strategies for large‑scale model training and inference.
This article explores how Remote Direct Memory Access (RDMA) technology transforms high‑performance computing, artificial intelligence, and cloud storage by eliminating data copies, bypassing the kernel, and offloading protocols to hardware, while reviewing key metrics, product ecosystems, real‑world use cases, challenges, and future trends.
Volcengine’s vRDMA brings high‑performance, low‑latency RDMA acceleration to cloud VPCs, combining self‑developed congestion control, elastic ENI integration, and compatibility with HPC, AI, and big‑data workloads to deliver up to 320 Gbps bandwidth and microsecond‑level latency.
The article provides an in‑depth technical overview of Remote Direct Memory Access (RDMA), covering its zero‑copy, kernel‑bypass, and protocol‑offload features, hardware and software ecosystems, and its impact on high‑performance computing, artificial intelligence, cloud storage, finance, and edge computing.
The article explains how distributed AI training performance depends on reducing inter‑card communication latency, introduces RDMA technology and its implementations (InfiniBand, RoCEv2, iWARP), compares their latency and scalability against traditional TCP/IP, and outlines the hardware components and trade‑offs of InfiniBand and RoCEv2 networks.
The whitepaper explains that AI model training and inference rely on massive data computation, with model sizes reaching billions of parameters, demanding low‑latency, high‑bandwidth, stable, scalable, and manageable networks; it compares RDMA‑based InfiniBand and RoCE solutions and offers design recommendations for future AI compute clusters.
This article provides a comprehensive technical analysis of InfiniBand architecture, its protocol stack, comparison with Ethernet‑based RDMA solutions like RoCE and iWARP, and an overview of Omni‑Path, highlighting performance advantages, design trade‑offs, and practical limitations.
This article surveys current network load‑balancing approaches—including CONGA, Hula, DRILL, Hermes, MP‑RDMA, ConWeave, Proteus, and CAVER—detailing their granularity, information exchange, signaling methods, and the performance gains they achieve in modern data‑center environments.
This article analyzes how the traditional OSI/TCP‑IP model is giving way to RDMA in high‑performance computing, compares Ethernet, InfiniBand and RoCE, evaluates leaf‑spine versus three‑tier data‑center designs, and examines NVIDIA SuperPOD architectures with detailed technical metrics.
This article provides an in‑depth technical analysis of DeepSeek's open‑source 3FS distributed file system, focusing on the StorageService architecture, space pooling, allocation mechanisms, reference counting, fragmentation handling, and the RDMA‑based read/write data path.
This article dissects DeepSeek's 3FS parallel file system, detailing its four‑component architecture, high‑throughput RDMA networking, metadata handling with FoundationDB, client access methods, chain replication (CRAQ), custom FFRecord format, and recovery mechanisms, offering a deep technical perspective for storage engineers.
This article dives deep into DeepSeek's 3FS parallel file system, detailing its four-component architecture, RDMA‑based high‑speed networking, client options, metadata and storage services, replication protocols, dynamic stripe sizing, and recovery mechanisms that enable efficient AI model training and inference.
This guide explains how to set up Baidu Baige's PD‑separated deployment for the DeepSeek R1 large‑language model, covering resource preparation, data acquisition, Prefill and Decode service configuration, and API invocation to achieve lower latency and higher throughput.
This article provides an in‑depth analysis of DeepSeek's open‑source 3FS distributed storage system, focusing on its network communication module, RDMA‑based design, core classes such as IBSocket, Listener, and IOWorker, and how these innovations advance high‑performance AI infrastructure.
This article explains the design and implementation of GPU cloud servers, covering data processing pipelines, hardware selection, topology, interconnect technologies, virtualization, multi‑GPU communication methods, and Baidu's practical solutions for both virtualized and bare‑metal instances to boost AI inference and training performance.
This article explores the hardware and networking foundations for training massive AI models, detailing the challenges of large‑scale RDMA deployment, the evolution of congestion‑control algorithms like DCQCN, TIMELY, HPCC, and AWS's SRD, and how hardware offload and programmable switches enable scalable, low‑latency AI infrastructure.
RoCE, originally built for InfiniBand, was adapted to Ethernet as RoCEv2, which uses IP/UDP headers to enable L3 routing but is highly sensitive to packet loss, requiring a lossless network and employing technologies such as PFC, ECN, DCQCN, and multi‑path transmission to maintain high RDMA performance.
This article explains the principles of Direct Memory Access (DMA) and Remote Direct Memory Access (RDMA), compares them with traditional TCP I/O, outlines RDMA’s features, protocol standards, communication pathways, queue mechanisms, and provides example code for setting up RDMA connections using RoCEv2.
InfiniBand RDMA, designed to network server buses, offers high bandwidth and ultra‑low latency through zero‑copy, kernel‑bypass communication, with a layered architecture (L1‑L5) and hardware components like Quantum‑2 Switch, ConnectX‑7 RNIC, and SHARP acceleration, supported by the Verbs API and OFED stack.
This article explains the fundamentals of physical networking layers, introduces cloud networking concepts such as VPC, overlay and underlay, and discusses how high‑performance technologies like RDMA and ACCL are evolving to support AI large‑model training in modern cloud environments.
This article compares three high‑performance distributed storage systems—Ceph, Alibaba's Pangu 2.0, and XSKY INFINI—examining their architectures, key technologies such as RTC thread models, append‑only writes, kernel‑bypass, RDMA, data compression, and metadata management to reveal how they exploit modern flash hardware.
This article explains RDMA technology, covering its core principles, programming model with Verbs API, various communication modes, and its impact on data‑center networking, high‑performance computing, and distributed storage, highlighting its low‑latency, zero‑copy advantages over traditional TCP/IP.
This article provides a detailed technical analysis of multi‑GPU server architectures, covering component breakdowns, NVSwitch networking, bandwidth calculations, and the differences between NVIDIA A100, A800, and H100 configurations for large‑scale AI workloads.
AI hardwareGPU architectureHigh-Performance Computing
This article provides a comprehensive technical analysis of InfiniBand’s protocol layers, topology, and performance advantages, compares Omni‑Path’s architecture, explains RDMA fundamentals, and details Ethernet‑based RDMA protocols such as RoCE and iWARP, highlighting their trade‑offs and use cases.
This tutorial walks through the complete RDMA programming workflow using the ibverbs API, covering device initialization, memory registration, completion queue and queue pair creation, state transitions, send/receive operations, completion handling, and resource cleanup with concrete C code examples.
Establishing an RDMA connection requires exchanging key parameters such as LID, QP number, and memory keys, and this article systematically outlines the essential information, compares six exchange methods—from static configuration to distributed services—and evaluates their advantages, drawbacks, and suitable scenarios.
This guide walks you through the hardware requirements, cost considerations, and step‑by‑step configuration of a simulated RDMA network using Soft‑RoCE on both Red Hat/CentOS and Ubuntu, including verification with ibv tools and performance testing.
This article explains the fundamentals of RDMA, compares libibverbs and rdma_cm with the user‑space rsocket API, and walks through a complete Go implementation using the smallnest/rsocket library, including both server and client code examples and practical deployment tips.
This article examines the evolution from the OSI and TCP/IP models to RDMA‑based technologies, compares traditional three‑tier and leaf‑spine architectures, analyzes NVIDIA SuperPOD designs, and evaluates Ethernet, InfiniBand, and RoCE switches to guide high‑throughput, low‑latency data‑center networking decisions.
This article explains the motivations behind RDMA, describes its architecture, key components, and protocols such as RoCEv2, and discusses future technical challenges for scaling RDMA in large AI and HPC data‑center networks.
The talk outlines ApsaraMQ’s journey to a fully serverless, cloud‑native messaging platform, detailing its compute‑storage separation, stateless proxy functions, RDMA‑enhanced performance, elastic scaling mechanisms, and how its event‑driven architecture empowers real‑time AI applications through seamless data vectorization.
This article examines the two dominant AI compute network architectures, InfiniBand and RoCEv2, detailing their designs, flow‑control mechanisms, performance, cost and scalability characteristics, and evaluates their respective advantages and limitations to guide network selection for AI data centers.
The article explains how RDMA technologies—including InfiniBand, RoCE, and iWARP—provide high‑throughput, low‑latency, CPU‑free data transfer for massive generative AI model training, compares their architectures, and discusses modern network designs and load‑balancing strategies to optimize AI‑focused data‑center networks.
The article examines the ultra‑large‑scale, high‑bandwidth, low‑latency, and automated network infrastructure needed for training generative AI models, covering custom network designs, congestion control, deterministic RDMA, topology choices such as Fat‑Tree, and emerging deterministic networking technologies.
This article provides a detailed technical analysis of Meta's SIGCOMM paper on RDMA over Ethernet for large‑scale AI training, examining the physical network deployment, congestion‑control mechanisms, topology choices, routing strategies, hardware design, and the practical challenges that remain.
The article analyzes how the rapid shift of data‑center spending toward AI/ML has accelerated RDMA and RoCE adoption, outlines market forecasts through 2028, explains the technical advantages of direct memory access, and examines the evolving server, NIC, and backend‑network landscapes that will shape future AI workloads.
The article explains how RDMA and RoCE technologies, originally built for high‑performance computing, are rapidly expanding in AI/ML data centers, driving massive market growth, faster GPU communication, and lower job completion times as server designs evolve toward higher GPU counts and faster NICs.
Baidu's HPN network solves hash‑collision bottlenecks in large‑model training by combining TOR‑affinity scheduling with Dynamic Load Balancing on self‑developed switches, boosting physical network bandwidth efficiency to about 95%, improving throughput by roughly 10% and adding a further 1.5% training‑speed gain via the BCCL library.
The article analyzes how AI‑driven elephant flows overload traditional Ethernet networks, causing long‑tail latency and victim‑flow congestion, and explains how adaptive routing, RDMA/ RoCE features, advanced congestion‑control algorithms, and high‑capacity switch chips can mitigate these challenges.
This article examines the impact of GPU communication bottlenecks on large‑model training, analyzes hash‑collision issues in high‑performance networks, and presents three practical solutions—including increasing RDMA streams, affinity‑aware scheduling, and dynamic load balancing—to boost effective network bandwidth up to 95%.
This article provides a detailed comparison of InfiniBand and RoCEv2 network architectures, examining their technical features, flow‑control mechanisms, performance, cost, and suitability for AI compute environments to guide designers in selecting the optimal solution.
This article explains the evolution from SATA and SAS to NVMe, details the NVMe over Fabrics (NVMeoF) architecture, compares TCP, FC‑NVMe and RDMA transports, and shows how these technologies reshape high‑performance SAN and data‑center storage solutions.
With AI models growing to billions of parameters, the choice of high‑performance interconnect—InfiniBand or RoCEv2—directly impacts training speed, scalability, latency, and operational complexity, and this article analyzes their architectures, performance metrics, vendor ecosystems, and suitability for large‑scale AI clusters.
The article provides a detailed technical analysis of RoCE (RDMA over Converged Ethernet), its two protocol versions, packet overhead, congestion‑control mechanisms, Soft‑RoCE implementation, and the challenges and performance implications of deploying RoCE in modern HPC environments compared to InfiniBand and traditional Ethernet solutions.
This article examines the evolution of network protocols from the OSI seven‑layer model and TCP/IP to RDMA technologies such as InfiniBand and RoCE, compares traditional three‑tier and leaf‑spine data‑center designs, and evaluates Ethernet, InfiniBand, and RoCE switches for high‑throughput, low‑latency HPC environments.
This article provides an in‑depth technical analysis of RoCE v2, covering its architecture, NIC requirements, and detailed comparisons with InfiniBand across physical layers, protocol stacks, switching, congestion handling, routing, and topology, while also highlighting the UEC alliance’s new transport protocol initiative.
This article explains how to construct high‑performance RoCE v2 and InfiniBand networks within a cloud‑native Kubernetes environment, detailing the underlying technologies, required components, configuration steps, and performance test results that demonstrate significant communication speed improvements for large‑scale AI model training.
This article explains how Remote Direct Memory Access (RDMA) technologies such as InfiniBand and RoCE bypass OS kernels to achieve ultra‑low latency and high bandwidth, discusses their hardware implementations, cost considerations, and their critical impact on large‑scale AI model training and HPC network design.
The article reviews ten years of RDMA development, tracing its origins, the rise and pitfalls of RoCEv1/v2, alternative approaches like iWARP and Cisco usNIC, and recent modernizations such as AWS SRD, Google Falcon and UltraEthernet, highlighting why protocol design choices have repeatedly stalled industry progress.
The rapid growth of AI model sizes has created unprecedented demands on network bandwidth, latency, stability, and automation, making efficient RDMA‑based interconnects, advanced congestion control, and intelligent deployment essential for scaling distributed training clusters to thousands of GPUs.
This article explains how the integration of sriov-network-operator and Kube-OVN automates the complex configuration and persistence of RDMA and SR‑IOV in Kubernetes, enabling high‑availability, multi‑tenant networking for AI distributed training workloads.
The talk examines the challenges of using standard Linux TCP for high‑performance data‑center workloads, explores how RDMA can provide zero‑copy and asynchronous kernel bypass, and presents experimental results from an FPGA‑based prototype that approaches 800 Gbps packet rates while highlighting congestion‑control and CPU‑utilization trade‑offs.
This article provides a comprehensive overview of InfiniBand technology, covering its history, architecture, packet format, layer functions, switching mechanisms, and performance advantages over Ethernet, while highlighting its rapid growth and future prospects in HPC environments.
This article explains how the shift from HDD to ultra‑fast SSDs and NVMe changes storage networking, compares NVMe with legacy SCSI, details NVMe‑oF transport options (FC, TCP, RDMA), examines RDMA variants, and outlines the network requirements and trade‑offs for deploying NVMe‑oF in modern data centers.
The talk explains how Vivo leverages a Kubernetes‑based solution that combines Calico and RoCEv2 to migrate offline training workloads from single‑NIC to multi‑NIC, integrating loss‑less RDMA, planning topology and IP allocation, and employing Volcano, SpiderPool, Macvlan, and Multus CNI for efficient container networking.
This article explains how network protocols and switch architectures—including OSI layers, TCP/IP, RDMA, InfiniBand, RoCE, and leaf‑spine designs—support high‑throughput, low‑latency AI and HPC workloads, compares Ethernet and InfiniBand markets, and examines NVIDIA’s Spectrum/X and SuperPOD solutions.
This article explains the differences between RDMA (including RoCE, InfiniBand, and iWARP) and traditional TCP/IP, describes the three main RDMA network types, compares their characteristics, and outlines their typical usage in distributed storage environments.
The article analyzes the 2022‑2027 global and Chinese network switch market sizes, outlines the competitive landscape dominated by Cisco and domestic players, and examines emerging technology trends such as lossless data‑center solutions, white‑box switches, silicon‑photonic optics, liquid‑cooling, TSN, and the coexistence of Ethernet and InfiniBand switches.
This article explains how the emergence of ultra‑fast SSDs and NVMe reshapes storage architecture, details the NVMe protocol and its extensions over fabrics, compares RDMA‑based transport options such as FC, TCP, and RoCE, and discusses network requirements and performance trade‑offs for modern data‑center deployments.
This article examines the architecture of AI compute clusters, explaining offline training and inference pipelines, the role of RDMA, and the technical differences between InfiniBand and RoCEv2—including latency, bandwidth, scalability, cost, and vendor considerations—to help engineers choose the optimal high‑performance network for large‑model training.
This article explains how AI and high‑performance computing drive the evolution of network protocols and data‑center switch designs, covering OSI basics, TCP/IP limitations, RDMA technologies, leaf‑spine topologies, Nvidia Spectrum platforms, and current market trends.
ByteFUSE, a user‑space FUSE‑based solution for ByteNAS, delivers low‑latency, high‑throughput, POSIX‑compatible storage across AI training, database backup, and search services by replacing NFS with a cloud‑native architecture that leverages CSI, RDMA, and kernel‑module hot‑upgrade techniques.
This article explains the fundamentals of network protocols and the OSI model, describes how high‑performance computing and AI workloads drive the transition from TCP/IP to RDMA technologies such as InfiniBand, RoCE and iWARP, and examines modern data‑center switch architectures, market trends, and NVIDIA’s AI‑focused networking solutions.
This article introduces NVIDIA's Quantum‑2 InfiniBand solution for high‑performance computing, explains its HDR 200 Gb/s architecture, and provides a comprehensive Q&A covering cable compatibility, SuperPod networking, UFM management, PCIe bandwidth, and RDMA support for both IB and Ethernet environments.
This article explains the fundamentals of Direct Memory Access (DMA) and Remote Direct Memory Access (RDMA), compares their data transfer mechanisms with traditional networking, and outlines RDMA's advantages, protocols, ecosystem, and real‑world adoption in high‑performance computing and data centers.
This article explains the principles of DMA and RDMA, compares RDMA protocols such as InfiniBand, RoCE, and iWARP, outlines their performance advantages, and reviews the key standards bodies, open‑source communities, hardware vendors, and real‑world adoption in high‑performance data centers.
This article provides a comprehensive overview of InfiniBand technology, covering its history, architecture, packet structure, layer hierarchy, switching mechanisms, and performance advantages over Ethernet, highlighting its role as a low‑latency, high‑bandwidth solution for high‑performance computing.
This article examines the RoCE protocol—an RDMA‑enabled Ethernet technology—its evolution, technical details, congestion‑control mechanisms, performance comparisons with InfiniBand, practical deployment issues in HPC clusters, and real‑world case studies such as Slingshot and application benchmarks.
This article provides an in‑depth examination of InfiniBand, covering its rapid development as a high‑bandwidth, low‑latency interconnect technology, the InfiniBand Trade Association, detailed packet structures, layered architecture, switching mechanisms, and a comparative analysis with Ethernet, highlighting its advantages for high‑performance computing.
Tencent’s Star Network delivers a 1.6 Tbps Ethernet‑RDMA fabric, fat‑tree topology supporting up to 4 K GPUs, multi‑track traffic aggregation and adaptive heterogeneous links plus a custom TCCL library, cutting AllReduce overhead from 35 % to 3.7 %, speeding AI training iterations by 32 % while automating deployment and providing sub‑second self‑healing.
The 2022 DPU Development Analysis Report outlines the evolution of Data Processing Units from CPU/NP and FPGA‑CPU architectures to ASIC‑CPU designs, discusses RDMA high‑speed networking, data‑plane forwarding techniques, network programmability, and the emerging open DPU software ecosystem, highlighting their performance, power, and cost implications for modern data centers.
The article outlines the evolution of data‑center architectures, identifies capacity, efficiency, and performance challenges of traditional storage‑compute models, and presents the emerging Diskless architecture—leveraging DPU, CXL, RDMA, and high‑throughput networking—to achieve decoupled, pool‑based resources and improve overall data‑center utilization.
The whitepaper outlines the evolution and challenges of modern data center architectures, introduces the Diskless architecture that decouples compute and storage using high‑density storage, DPUs, RDMA and CXL, and details its design principles, key technologies, and industry implementations.