What Optimizations Power DeepSeek’s High‑Efficiency LLMs?

Inference Acceleration

Grouped Query Attention (GQA) replaces Multi‑Head Attention (MHA). The subsequent Multi‑head Latent Attention (MLA) uses compressed projection matrices, dramatically reducing key‑value (KV) computation during inference.

Depth‑First Scaling

Model depth is increased rather than width: DeepSeek LLM 7B (30 layers) → LLM 67B (95 layers).

Dataset Scaling‑Law Impact and MoE

Cross‑dataset training can invalidate the expected scaling law, motivating the adoption of a Mixture‑of‑Experts (MoE) architecture.

DSMoE Enhancements

DSMoE improves segmentation and shared‑expert isolation, substantially increasing balanced computation across experts.

Auxiliary‑Loss‑Free Load Balancing

Gate scores for each expert are adjusted via bias updates, achieving load balance without introducing auxiliary loss gradients.

Convergence Acceleration

Training uses AdamW with beta1=0.9, beta2=0.95, weight_decay=0.1 and a three‑stage warmup‑and‑step‑decay scheduler: warmup for 2 k steps, main phase on 80 % of the dataset, tail phase on the final 90 % of the dataset.

HAI‑LLM 4D Parallel Framework

Combines data, tensor, pipeline, and 1F1B pipeline parallelism. Integrates Zero‑Bubble concepts and evolves into DualPipe dual‑pipeline optimization, which improves all‑to‑all dispatch and combine operations and raises hardware utilization compared with LLaMA‑style clusters.

DPO Replaces PPO (GRPO)

Group Relative Policy Optimization (GRPO) removes the value network, reducing computation. Advantage is estimated via grouped sampling.

8‑Bit Floating‑Point Quantization

Both model parameters and KV caches are quantized to 8‑bit, enabling high‑precision FP8 GEMM. A CUDA‑based quantization framework implements the conversion.

Multi‑Token Prediction (MTP)

MTP extends context length while keeping compute cost low. The model learns serial head prediction, achieving chain‑learning behavior. Head and embedding layers are shared to limit parameter growth.

Compute‑to‑Memory Trade‑offs

Recompute RMSNorm and MLA up‑projection instead of storing them.

Use Exponential Moving Average to approximate scaling‑law parameters.

Share head and embedding parameters, especially in the MTP architecture.

Distillation Reinforcement Pipeline

Three‑step process:

Cold start with chain‑knowledge extraction.

Pre‑reinforcement using high‑quality logical data.

Selective fine‑tuning.

Leverages GRPO reward signals, chain‑of‑thought prompts, multi‑model distillation, and sampling‑based learning.

Recall Testing

Needle‑In‑A‑Haystack (NIAH) tests are used to evaluate retrieval performance.