DeepSeek's Three Pillars: MoE, MLA, and Multi-Token Prediction Demystified

On 20 January 2025, DeepSeek launched the DeepSeek chatbot—based on the DeepSeek-R1 model—free for iOS and Android. By 27 January, DeepSeek surpassed ChatGPT as the most downloaded freeware app on the iOS App Store in the United States, triggering an 18% drop in Nvidia's share price.

Overall, DeepSeek stands on the shoulders of giants and delivers three important innovations:

DeepSeekMoE

Multi-Head Latent Attention (MLA)

Multi-Token Prediction (MTP)

1. DeepSeekMoE

Mixture-of-Experts (MoE) is not invented by DeepSeek, but DeepSeek adapts it to the Transformer architecture, targeting the feed-forward network (FFN) layers.

Because dense FFN layers incur high computation, the motivation for MoE is to reduce compute cost and energy consumption through sparse activation .

The left side of the diagram shows traditional MoE: the dense FFN is split into N smaller expert networks. A router selects K experts for each token, computes their outputs, and combines them via weighted sum. Experts maintain the hidden dimension directly, avoiding the up-projection and down-projection of a standard FFN, thus reducing computation.

DeepSeek researchers found that traditional MoE uses too few experts, making each expert insufficiently specialized. Their first attempt doubles the number of experts to 2N while halving each expert's parameter count to keep total compute constant (middle diagram).

They further optimize by adding a shared expert that handles common capabilities required by all tokens (right diagram).

Experiments show that at equal compute budget, DeepSeekMoE significantly outperforms both dense networks and traditional MoE .

2. Multi-Head Latent Attention

LLaMA's Grouped-Query Attention (GQA) is a downgraded optimization, whereas DeepSeek's MLA reduces attention computation complexity and memory overhead while preserving model expressiveness .

The principle of MLA is straightforward: compress the embedding vectors into low-dimensional latent representations for keys and values. When needed, decompress them back to original dimension. This reduces the KV-Cache storage requirement.

Rotary Positional Encoding (RoPE) poses a challenge because compressed vectors cannot directly add positional information. DeepSeek solves this by appending extra dimensions to Q and K vectors that encode positional information, while V vectors remain unchanged. As shown in the diagram, Q and K gain identical extra dimensions for position.

3. Multi-Token Prediction

The first two innovations primarily optimize inference, while MTP improves training efficiency and accelerates model convergence by predicting multiple future tokens simultaneously .

References:

1.

https://arxiv.org/abs/2401.06066

2.

https://arxiv.org/abs/2405.04434

3.

https://arxiv.org/abs/2412.19437