What Makes DeepSeek’s New Architecture a Game‑Changer? Inside MLA, GRPO, and MoE Innovations

DeepSeek Core Architectural Innovations

DeepSeek builds on the standard Transformer decoder stack but introduces three key components to improve inference efficiency and alignment.

Multi‑Head Latent Attention (MLA)

MLA, first presented in DeepSeek V2, rewrites the attention operator to compress the key‑value (KV) cache. Each query‑KV pair is reduced by 93.3 %, allowing many more tokens to be cached in the same GPU memory and increasing throughput during generation.

Group Relative Policy Optimization (GRPO)

GRPO replaces a large‑scale reward model with a group‑wise baseline estimator. For each input the old policy π_old samples a set of outputs {o₁,…,o_G}. The training objective maximizes the expected reward while subtracting the group baseline, eliminating the need for a separate evaluator model.

The reward model combines three signals:

Accuracy reward : verifies exact correctness (e.g., math answers, LeetCode test cases).

Format reward : encourages the model to wrap reasoning in predefined tags, improving output structure.

Language‑consistency reward : promotes coherent phrasing.

Mixture‑of‑Experts (MoE) Optimizations

Instead of a few large experts, DeepSeek uses many tiny experts plus a shared expert. Load balancing is achieved without an auxiliary loss: a bias term is added to the routing logits and adjusted dynamically (not learned by gradient descent) to ensure each expert receives a balanced token count.

Experts are divided into:

Shared experts that are always routed.

Routing experts whose load is balanced by the dynamic bias.

Training Methodology

DeepSeek‑R1 follows a three‑stage pipeline:

Language‑model pre‑training : train on massive web data to predict the next token, producing a base decoder model.

Supervised fine‑tuning (SFT) : fine‑tune on instruction‑following data to improve usefulness.

Preference tuning : apply reinforcement‑learning (RL) with human‑feedback‑style rewards to align outputs with user preferences.

A special intermediate model, DeepSeek‑R1‑Zero , is trained purely by RL (no SFT data). It generates high‑quality chain‑of‑thought (CoT) examples that are filtered and used as “cold‑start” data for the final R1 model, reducing reliance on large manually annotated datasets.

RL details :

Generate Python sorting tasks, then automatically verify correctness with a linter, execution, and unit‑test generation.

Collect reward signals for accuracy, format compliance, and execution efficiency.

Iteratively update the policy based on these signals, improving both reasoning and non‑reasoning tasks.

Model Variants and Performance

Two variants are released:

DeepSeek‑R1‑Zero : RL‑only model excelling at long‑chain reasoning.

DeepSeek‑R1 : Multi‑stage model that retains strong reasoning while addressing language mixing and readability.

Benchmarks show that DeepSeek‑R1 matches or exceeds GPT‑4o‑Mini on intelligence and consistency scores and outperforms Llama 3.1 by roughly a factor of two on reasoning‑heavy tasks. Safety testing reveals occasional harmful outputs in adversarial scenarios, indicating room for improvement.

Scale and Parameters

The architecture consists of 61 decoder layers: the first three are dense, and the remaining layers are MoE. Parameter counts range from 1.5 B (distilled) to 32 B (full), offering trade‑offs between cost and performance.

References

[1]

DeepSeek – https://zhida.zhihu.com/search?content_id=253172569&content_type=Article&match_order=1&q=DeepSeek [2] MLA – https://zhida.zhihu.com/search?content_id=253172569&content_type=Article&match_order=1&q=MLA [3] GRPO – https://zhida.zhihu.com/search?content_id=253172569&content_type=Article&match_order=1&q=GRPO [4] DeepSeek‑R1 – https://zhida.zhihu.com/search?content_id=253172569&content_type=Article&match_order=1&q=DeepSeek-R1 [5] DeepSeek‑R1‑Zero – https://zhida.zhihu.com/search?content_id=253172569&content_type=Article&match_order=1&q=DeepSeek-R1-Zero [6] MoE – https://zhida.zhihu.com/search?content_id=253172569&content_type=Article&match_order=1&q=MoE [7] OpenAI O1 – https://zhida.zhihu.com/search?content_id=253172569&content_type=Article&match_order=1&q=OpenAI+O1 [8] Llama 3.1 – https://zhida.zhihu.com/search?content_id=253172569&content_type=Article&match_order=1&q=Llama3.1