Tagged articles

Multi-Head Latent Attention

6 articles · Page 1 of 1

Oct 25, 2026 · Artificial Intelligence

Attention Algorithms 2017-2026: MHA, GQA, MLA, NSA Selection Guide

This comprehensive survey covers attention mechanisms from 2017's Scaled Dot-Product Attention to 2026's NSA and FlashAttention-4, organizing them into four optimization tracks—KV cache compression, exact attention acceleration, long-context extension, and frontier alternative architectures—with concrete benchmarks, trade-offs, and practical selection criteria for each use case.

FlashAttentionHybrid ArchitecturesKV Cache

0 likes · 12 min read

Attention Algorithms 2017-2026: MHA, GQA, MLA, NSA Selection Guide

Machine Learning Algorithms & Natural Language Processing

Oct 11, 2026 · Artificial Intelligence

DeepSeek's Cost Curve Chronicle: How Token Economics Rewrote LLM Architecture

This article chronicles DeepSeek's paper timeline from 2023 to 2026, analyzing how each model generation optimized training compute, KV cache, and throughput via MoE, MLA, sparse attention, and conditional memory, shifting long-context competition from maximum length to per-token byte cost.

DeepSeekGRPOKV-Cache Optimization

0 likes · 37 min read

DeepSeek's Cost Curve Chronicle: How Token Economics Rewrote LLM Architecture

PaperAgent

Jan 21, 2026 · Artificial Intelligence

Inside DeepSeek’s FlashMLA Update: What’s New in the MODEL1 Architecture

DeepSeek’s recent FlashMLA update introduces the new MODEL1, featuring a tighter KV-Cache layout, an extra two-stage cache, and a fixed 512×512 head dimension, with four code changes detailed in a public GitHub commit and illustrated by comparative diagrams.

AI architectureDeepSeekFlashMLA

0 likes · 3 min read

Inside DeepSeek’s FlashMLA Update: What’s New in the MODEL1 Architecture

Code Archaeology

Oct 24, 2025 · Artificial Intelligence

DeepSeek's Three Pillars: MoE, MLA, and Multi-Token Prediction Demystified

This article explains DeepSeek's three core innovations—DeepSeekMoE with fine-grained experts and shared experts, Multi-Head Latent Attention (MLA) for compressed KV-cache, and Multi-Token Prediction (MTP) for faster training convergence—detailing how each reduces compute costs and memory overhead while maintaining model performance.

DeepSeekKV-Cache OptimizationLLM Architecture

0 likes · 5 min read

DeepSeek's Three Pillars: MoE, MLA, and Multi-Token Prediction Demystified

Network Intelligence Research Center (NIRC)

Mar 26, 2025 · Artificial Intelligence

Enable Traditional LLMs to Use DeepSeek’s Multi‑Head Latent Attention Without Retraining

The paper introduces MHA2MLA, a data‑efficient fine‑tuning framework that converts pre‑trained multi‑head attention LLMs to DeepSeek’s Multi‑Head Latent Attention architecture, achieving up to 92% KV‑cache compression with less than 0.5% performance loss on long‑context tasks.

LLMLow-Rank ApproximationMulti-Head Attention

0 likes · 8 min read

Enable Traditional LLMs to Use DeepSeek’s Multi‑Head Latent Attention Without Retraining

Baobao Algorithm Notes

May 9, 2024 · Artificial Intelligence

Inside Deepseek‑V2: How Multi‑Head Latent Attention Cuts KV‑Cache and Boosts Performance

This article provides an in‑depth technical analysis of Deepseek‑V2, covering its 236B parameter size, Multi‑Head Latent Attention optimization that reduces KV‑cache memory, architectural details, training pipelines, infrastructure choices, and performance results on benchmarks such as MMLU and instruction following.

AI architectureDeepSeekMulti-Head Latent Attention

0 likes · 17 min read

Inside Deepseek‑V2: How Multi‑Head Latent Attention Cuts KV‑Cache and Boosts Performance