How DeepSeek-V3 Achieves Massive Scale with FP8, MoE, and System Optimizations

Model Architecture

DeepSeek‑V3 follows a system‑algorithm co‑design approach and retains the MLA (Memory‑Light Attention) and Mixture‑of‑Experts (MoE) structures introduced in V2.

MLA technique : MLA compresses the KV cache dimensions in a LoRA‑like fashion and moves the up‑projection to the query (Q) and output (O) matrices, eliminating repeated decompression. This reduces KV‑cache per‑token memory at the cost of added system complexity and without a clear advantage over MQA.

MoE structure : Instead of a few large experts, V3 employs many small experts (256 total). The model contains 671 B total parameters but only 37 B active (expert‑selected) parameters, compared with V2’s 236 B total and 21 B active parameters. The higher sparsity lowers FLOP‑per‑parameter cost.

Training efficiency improves to roughly 180 K GPU‑hours per trillion tokens (V2: 172.8 K), fulfilling the “Economical” claim of the V2 report.

Additional innovations include an auxiliary‑loss‑free load‑balancing strategy for MoE experts and multi‑token prediction (MTP) , which supplies richer supervision during training and enables speculative sampling during decoding.

Training Optimizations

DeepSeek‑V3 is the first open‑source large MoE model trained with FP8 mixed‑precision . To mitigate FP8 overflow and MoE instability, the model adopts a uniform E4M3 format and applies fine‑grained quantization:

Per‑tile quantization of size 1×128.

Per‑group quantization of size 128×128.

This reduces quantization error and approximates micro‑scaling formats, though current hardware lacks native support, requiring partial‑sum implementations for FP8 matrix multiplication.

Memory savings are achieved by storing optimizer states in BF16 and selectively recomputing expensive operations such as RMSNorm, MLA up‑projection, and SwiGLU. These reductions enable more aggressive parallelism.

Parallelism strategy :

64‑way expert parallelism.

16‑way pipeline parallelism.

ZeRO‑1 data parallelism.

All‑to‑all communication introduced by expert parallelism is mitigated through group routing , limiting each token to activate experts on only four nodes, halving cross‑node traffic. System‑level pipelining overlaps intra‑node and inter‑node communication, keeping the communication‑to‑computation ratio near 1:1 and allowing concurrent scheduling of forward and backward micro‑batches.

Pipeline parallelism uses a bidirectional pipeline (similar to Chimera) instead of the common interleaved 1F1B schedule, reducing pipeline bubbles while still overlapping forward and backward passes.

Inference Optimizations

Deploying MoE models efficiently requires a DP+EP (data‑parallel + expert‑parallel) strategy to avoid the dense‑model inference path that erodes MoE benefits during decoding.

PD‑separation handles prefilling and decoding separately:

Prefill stage : Attention uses 4‑way tensor parallelism combined with 8‑way data parallelism; the MoE module runs with 32‑way expert parallelism to meet first‑token latency targets.

Decode stage : Expert parallelism scales to 320 ways (256 small experts + 64 hot experts), reducing decode latency and alleviating load imbalance.

To hide the latency of all‑to‑all communication, V3 adopts the NanoFlow double‑stream inference technique, executing computation and communication for different micro‑batches concurrently, thereby improving device utilization.

References

How to view DeepSeek’s MoE large model DeepSeek‑V2? https://zhihu.com/question/655172528/answer/3504750755

MLA increases attention head count to compensate for precision loss; similar techniques can be applied to MQA. Fair comparison is lacking.

Auxiliary‑Loss‑Free Load Balancing Strategy for Mixture‑of‑Experts, https://arxiv.org/abs/2408.15664

Using FP8 with Transformer Engine, https://docs.nvidia.com/deeplearning/transformer-engine/user-guide/examples/fp8_primer.html

Microscaling Data Formats for Deep Learning, https://arxiv.org/abs/2310.10537

Chimera: Efficiently Training Large‑Scale Neural Networks with Bidirectional Pipelines, https://arxiv.org/abs/2107.06925

NanoFlow: Towards Optimal Large Language Model Serving Throughput, https://arxiv.org/html/2408.12757v1