DeepSeek V3 Architecture Deep Dive: MoE, MLA, DualPipe, FP8 Mixed Precision & Multi‑Token Prediction

DeepSeek V3 Overview

DeepSeek, founded in July 2023, has released three large‑language‑model families: DeepSeek‑R1, DeepSeek‑V2.5 and DeepSeek‑V3. DeepSeek‑V3 contains 671 billion parameters (≈37 billion active per token) and adopts a Mixture‑of‑Experts (MoE) backbone to keep inference and training costs low while achieving state‑of‑the‑art performance.

Mixture‑of‑Experts (MoE) Architecture

In DeepSeek‑V3 the traditional Feed‑Forward Network (FFN) is replaced by a DeepSeek‑MoE layer. Each token is routed to a small subset of experts by a gating network. The MoE layer consists of many fine‑grained experts and a set of shared experts, which reduces knowledge redundancy across experts.

Each expert is a small neural sub‑network (often an FFN) that processes only the tokens assigned to it.

Routing is performed with a sigmoid‑based Top‑K selection, followed by a softmax‑like normalization.

Load‑Balancing without Auxiliary Loss

Instead of adding an auxiliary loss to force balanced expert usage, DeepSeek‑V3 introduces a bias term b_i for every expert. During training the bias is decreased by a hyper‑parameter γ for overloaded experts and increased by γ for under‑utilised experts. This dynamic bias directly adjusts the routing scores, achieving load balance without extra loss terms.

Multi‑head Latent Attention (MLA)

MLA compresses the Key and Value matrices before they are cached. A low‑rank factorisation (e.g., a 4×2 projection matrix) reduces the KV‑cache size and memory bandwidth. The compressed representation is later expanded to the original dimension for the standard attention computation. This compression yields faster inference with negligible accuracy loss.

DualPipe Pipeline Parallelism

Training is organised into four stages per compute block: (1) attention computation, (2) all‑to‑all communication, (3) MLP computation, and (4) a second all‑to‑all communication. DualPipe overlaps the communication steps with the computation steps using warp‑level specialization, eliminating the “pipeline bubble” that appears in conventional pipeline parallelism. The design achieves near‑zero all‑to‑all overhead even when communication occurs at every token.

Intra‑node communication uses NVLink; inter‑node communication uses InfiniBand.

A token is allowed to be processed by at most four nodes, limiting network load.

Warp specialization assigns different warps to send data over IB, forward data to NVLink, or receive data from NVLink, enabling simultaneous communication and computation.

Mixed‑Precision FP8 Training

DeepSeek‑V3 adopts a hybrid precision scheme:

FP8 (8‑bit floating point) is used for dense matrix multiplications (GEMM) and other compute‑intensive kernels, cutting memory usage to roughly 50 % of FP16 and roughly doubling raw compute throughput.

Sensitive modules—LayerNorm, attention, embeddings, and the MoE gating network—remain in BF16 or FP32 to preserve numerical stability.

Quantisation granularity:

Activations are quantised group‑wise (group size 1×128).

Weights are quantised block‑wise (block size 128×128).

During GEMM the intermediate results are accumulated in FP32 on CUDA cores before being written back, preventing overflow/underflow.

Multi‑Token Prediction (MTP)

Instead of the classic autoregressive single‑token prediction, MTP adds several parallel output heads that jointly predict the next 3–5 tokens in a single forward pass. A multi‑token loss jointly optimises the sequence, which speeds up generation and reduces perplexity on downstream tasks.

Inference Deployment

Inference is split into two stages:

Prefill stage : builds the KV‑cache for the prompt. It uses 4‑way tensor parallelism (TP4), sequence parallelism (SP) and 8‑way data parallelism (DP8). This stage runs on a cluster of 2048 NVIDIA H800 GPUs.

Decode stage : generates tokens one by one. It combines TP4, SP and 80‑way data parallelism (DP80) for the attention part, and 320‑way expert parallelism (EP320) for the MoE part. Redundant experts and a shared expert are deployed to keep the workload balanced. The decode stage runs on 40 nodes with a total of 320 GPUs.

Both stages employ the same communication stack: intra‑node NVLink, inter‑node InfiniBand, and the warp‑level specialization described in DualPipe. Redundant experts and dynamic routing further smooth load spikes, allowing each token to use on average 3.2 experts (up to 13 experts when scaling).

Parallel Configuration Example

// Example configuration (illustrative only)
 tp_size = 4      # tensor parallelism (TP4)
 dp_size = 8      # data parallelism for prefill (DP8)
 dp_decode = 80   # data parallelism for decode (DP80)
 ep_size = 32     # expert parallelism for MoE in prefill (EP32)
 ep_decode = 320  # expert parallelism for MoE in decode (EP320)
 num_gpus_per_node = 32

These settings illustrate how DeepSeek‑V3 combines tensor, sequence, data and expert parallelism to scale training on thousands of GPUs and to serve inference with high throughput and low latency.