Understanding DualPipe: DeepDive into DeepSeek‑R1 Architecture (Part 5)

The article is the fifth installment of a series that deeply explores the DeepSeek‑R1 model architecture. It begins by recalling earlier posts on Mixture‑of‑Experts (MoE) and Multi‑Head Latent Attention (MLA), noting that MoE reduces activation parameters (e.g., DeepSeek‑V3‑671B uses 37 B activations) and MLA cuts KV‑cache size by 93.3 % while speeding up inference.

It then introduces the DualPipe component, which aims to raise the compute‑communication ratio during training. Before diving into DualPipe, the article reviews the fundamentals of forward and backward propagation: forward passes transform inputs layer‑by‑layer, while backward passes compute loss gradients via the chain rule and update weights with optimizers such as Adam. It also lists basic facts, such as the order of operations for a single sample and the batch‑wise update pattern.

Next, the article discusses the challenges of large‑scale distributed training. When a model fits on a single GPU, data parallelism (e.g., PyTorch Distributed) suffices by replicating the model across GPUs and synchronising gradients. For trillion‑parameter LLMs, model parallelism is required, splitting the model across hundreds or thousands of GPUs (e.g., GPT‑3’s 175 B‑parameter model on 10 000 A100 GPUs). The text explains layer‑wise partitioning, tensor‑level splitting, and expert‑module distribution, and notes that data parallelism and model parallelism can be combined.

To illustrate pipeline parallelism, the article describes the 1F1B schedule from the PipeDream paper (2018) and the Zero Bubble Pipeline Parallelism (2023, Sea AI). Zero Bubble separates the gradient of inputs (B) from the gradient of weights (W), reducing sequential dependencies and overlapping communication with computation.

DualPipe builds on Zero Bubble with two main innovations:

Fine‑grained stage partitioning: each compute block is divided into four components—Attention, All‑to‑All communication, MLP, and All‑to‑All aggregation. For backward blocks, Attention and MLP are further split into input‑gradient (B) and weight‑gradient (W) sub‑stages.

Bidirectional pipeline scheduling: small batches are fed from both ends of the pipeline, allowing most communication to overlap completely. DualPipe maintains two copies of model parameters so that, for example, with eight devices running an eight‑layer model, device 0 holds both layer 0 and layer 7 parameters, and device 7 holds both layer 7 and layer 0 parameters, enabling mirrored parameters and two‑way data flow.

These design choices significantly improve hardware utilisation and training throughput. To support DualPipe, DeepSeek also implements a custom high‑efficiency all‑to‑all communication kernel that reduces the number of streaming multiprocessors dedicated to communication; further details are referenced in the DeepSeek‑V3 technical report.

The article concludes that DualPipe, together with extensive infrastructure optimisations, allows DeepSeek to fully exploit GPU clusters despite limited resources compared with larger labs, and hopes the explanation helps readers understand the scheduling mechanism.

References:

DeepSeekMoE: https://arxiv.org/pdf/2401.06066

DeepSeekV2: https://arxiv.org/abs/2405.04434

DeepSeekV3: https://arxiv.org/abs/2412.19437

DeepSeekR1: https://arxiv.org/abs/2501.12948

Meata MTP: https://arxiv.org/pdf/2404.19737