Inside Deepseek‑V2: How Multi‑Head Latent Attention Cuts KV‑Cache and Boosts Performance

Overview

Deepseek recently released the Deepseek‑V2 model, a 236B‑parameter mixture‑of‑experts (MoE) LLM that continues the technical direction of the January Deepseek‑MoE release. The model and its dialogue‑aligned version are fully open‑sourced under the MIT license and can be used commercially. For developers with limited compute, an API with the lowest market price is also offered.

Core Optimization – Multi‑Head Latent Attention (MLA)

MLA is introduced to alleviate KV‑cache memory pressure during long‑sequence generation. Traditional KV‑cache stores full key and value matrices, which can exhaust GPU memory for long contexts. MLA compresses the cache by projecting hidden states into a lower‑dimensional space before storing them, then reconstructs the full representations during attention computation.

"hidden_size": 5120,
"kv_lora_rank": 512,
"moe_intermediate_size": 1536,
"q_lora_rank": 1536,
"qk_nope_head_dim": 128,
"qk_rope_head_dim": 64

The forward pass first projects the hidden state (size 5120) to a 1536‑dimensional query space, then splits it into q_nope (128) and q_pe (64) components. A similar compression is applied to the key/value side, yielding a compressed KV matrix of only 576 dimensions.

# __init__
self.kv_a_proj_with_mqa = nn.Linear(
    self.hidden_size,
    config.kv_lora_rank + config.qk_rope_head_dim,
    bias=config.attention_bias,
)
self.kv_a_layernorm = DeepseekV2RMSNorm(config.kv_lora_rank)
self.kv_b_proj = nn.Linear(
    config.kv_lora_rank,
    self.num_heads * (self.q_head_dim - self.qk_rope_head_dim + self.v_head_dim),
    bias=False,
)

During attention, the compressed KV is split into a non‑rotary part ( k_nope) and a rotary part ( k_pe). The rotary part receives the YaRN‑extended positional encoding, while the non‑rotary part carries the original content. This decoupling allows the model to keep the benefits of RoPE without inflating cache size.

Architecture Details

The model consists of 60 decoder layers, each with 128 attention heads. It uses a shared‑expert MoE design: 160 experts per token, with 6 experts activated per token and 2 shared experts always active, resulting in 8 experts contributing to each token’s computation. The embedding dimension is 5120, and the vocabulary size is 102400.

DeepseekForCausalLM(
  (model): DeepseekModel(
    (embed_tokens): Embedding(102400, 5120)
    (layers): ModuleList(
      (0): DeepseekDecoderLayer(
        (self_attn): DeepseekAttention(
          (q_a_proj): Linear(5120, 1536, bias=False)
          (q_a_layernorm): DeepseekRMSNorm()
          (q_b_proj): Linear(1536, 24576, bias=False)
          (kv_a_proj_with_mqa): Linear(5120, 576, bias=False)
          (kv_a_layernorm): DeepseekRMSNorm()
          (kv_b_proj): Linear(5120, 32768, bias=False)
          (o_proj): Linear(163840, 5120, bias=False)
          (rotary_emb): DeepseekYarnRotaryEmbedding()
        )
        (mlp): DeepseekMLP(...)
        ...
      )
      ...
    )
    (norm): DeepseekRMSNorm()
  )
  (lm_head): Linear(5120, 102400, bias=False)
)

The architecture follows modern LLM conventions: pre‑norm residual connections, RMSNorm, SiLU activation, and bias‑free linear projections to benefit flash‑attention.

Training Pipeline

Length‑extrapolation using YaRN: the model is trained with YaRN‑based positional extensions, enabling a context window of up to 128k tokens.

Instruction alignment (SFT + RLHF): the model undergoes supervised fine‑tuning on dialogue data, followed by preference alignment using the GRPO algorithm, a resource‑efficient variant of PPO that updates only the policy model while keeping the critic static.

Infrastructure Optimizations

Training uses pipeline parallelism (pp=16) and expert parallelism across 8 nodes (160 experts total) without tensor parallelism, reducing communication overhead. Data parallelism is implemented with ZeRO‑1 to shrink optimizer state memory. The hardware stack includes NVLink/NVSwitch intra‑node and InfiniBand inter‑node networking, all orchestrated by the custom HAI‑LLM runtime.

To avoid load imbalance, Deepseek‑V2 introduces three balancing dimensions:

Expert‑level balance: ensures no single expert is over‑trained.

Machine‑level balance: distributes the six active experts per token across different machines.

Communication‑level balance: limits the frequency of any machine appearing in the expert‑assignment matrix, preventing hotspots.

Model Performance

Deepseek‑V2 achieves 236B parameters with a low activation footprint, delivering high inference speed. On the MMLU multiple‑choice benchmark it scores second overall, trailing only LLaMA‑3 70B, and surpasses the previous 67B non‑MoE Deepseek‑V1. Training cost is reduced compared to the dense baseline, and KV‑cache memory usage is dramatically lower, yielding higher throughput.

Qualitative evaluation shows strong instruction‑following ability and robust performance on Chinese‑dominant data (Chinese data proportion is 1.12× that of English).

Discussion

Key findings from the official report include:

Instruction fine‑tuning requires at least 10 000 examples; fewer examples cause a noticeable drop in IFEval scores, and scaling model size cannot compensate for insufficient data.

Human‑preference alignment improves open‑ended question answering but introduces an “alignment tax” that can reduce leaderboard scores; Deepseek‑V2 mitigates this with refined data processing and training tricks.

Online preference alignment outperforms offline methods in reinforcement learning stages.

Conclusion

Deepseek‑V2 integrates proven LLM training strategies—YaRN length extrapolation, GRPO‑based alignment, and the MLA‑driven KV‑cache compression—into a cohesive system that maximizes algorithmic efficiency, engineering scalability, and data utilization, resulting in a competitive open‑source LLM.