DeepSeek's Cost Curve Chronicle: How Token Economics Rewrote LLM Architecture

This technical chronicle traces the evolution of DeepSeek's large language model papers from October 2023 to September 2026, framing each release as a recalculation of three core cost metrics: training compute (GPU-hours), KV cache memory (bytes per token), and generation throughput (tokens per second). The author structures the timeline into five "acts" plus multimodal side lines, showing how architectural innovations — Mixture-of-Experts (MoE), Multi-head Latent Attention (MLA), Native Sparse Attention (NSA/DSA), Conditional Memory (Engram), and hybrid attention (CSA/HCA) — progressively reduced the per-token denominator, enabling longer contexts, larger models, and cheaper inference.

Act I: Laying the Foundation (2023-10 to 2024-02)

The first public weights were DeepSeek-Coder (arXiv:2401.14196), trained on 2T tokens with a 16K window and cross-file infilling as a pre-training objective. Simultaneously, DeepSeek LLM (arXiv:2401.02954) established scaling laws for 7B/67B dense models, and DeepSeekMoE (arXiv:2401.06066) introduced fine-grained expert splitting (mN experts, mK activated) plus Ks shared experts for common knowledge. A 2B MoE matched a 2.9B GShard model with 1.5× fewer parameters. DeepSeekMath (arXiv:2402.03300) continued pre-training on 120B math tokens and proposed GRPO — a PPO variant that replaces the value network with group-relative advantage estimation — which became the RL backbone for later models (Prover, R1).

Figure: DeepSeekMath's data pipeline — FastText recall from Common Crawl, annotator-discovered math domains. Source: DeepSeek-AI, DeepSeekMath (arXiv:2402.03300), official repo images/data_pipeline.png .

Act II: Architecture Delivers First Cost Numbers (2024-03 to 2024-08)

DeepSeek-V2 (arXiv:2405.04434) — 236B total / 21B active, 128K context, 8.1T pre-train tokens — combined DeepSeekMoE and MLA. MLA projects keys/values into a low-dimensional latent vector, caching only that vector during inference. Relative to the dense DeepSeek 67B baseline, V2 reported: −42.5% training cost , −93.3% KV cache , ×5.76 max generation throughput . The author notes MLA stabilised as the attention backbone; later attention changes (sparse, hybrid) are additive, not replacements. Two side lines appeared: DeepSeek-Prover (arXiv:2405.14333) for formal math in Lean 4, and Fire-Flyer AI-HPC (arXiv:2408.14158) documenting a 10K PCIe A100 cluster achieving near-DGX performance at half cost and 40% less power, with the 3FS distributed filesystem later open-sourced. DeepSeek-Coder-V2 (arXiv:2406.11931) continued pre-training from a V2 checkpoint for 6T more tokens, expanding language support to 338 and context to 128K, illustrating that domain models are downstream of the general backbone.

Figure: V2 Transformer block with MoE (shared + routed experts) and MLA (cached latent vector). Source: DeepSeek-AI, DeepSeek-V2 (arXiv:2405.04434), official repo figures/architecture.png .

Act III: V3 and R1 — Two Shifts at Once (2024-10 to 2025-01)

DeepSeek-V3 (arXiv:2412.19437) scaled to 671B total / 37B active, 14.8T tokens. Key engineering choices: bias-based load balancing (no auxiliary loss) and multi-token prediction objective. Training consumed 2.788M H800 GPU-hours with zero irreversible loss spikes or rollbacks — a stability claim rarely explicit in prior reports. DeepSeek-R1 (arXiv:2501.12948) argued reasoning can be elicited by pure RL on verifiable tasks, without human-annotated chains-of-thought; emergent behaviours (self-verification, strategy switching) appeared during training. Crucially, R1 showed large-model reasoning patterns can be distilled into smaller models , enabling a divide-and-conquer post-training strategy. Multimodal line: Janus (arXiv:2410.13848) decoupled understanding/generation visual encoders; JanusFlow (arXiv:2411.07975) added rectified flow in the language backbone; DeepSeek-VL2 (arXiv:2412.10302) ported MoE+MLA to vision-language with dynamic tiling for arbitrary aspect ratios (active params 1.0B/2.8B/4.5B).

Act IV: The Year of Sparsification (2025-02 to 2025-12)

Three sparsity axes advanced in parallel:

Context sparsity: Native Sparse Attention (NSA) (arXiv:2502.11089) — hierarchical coarse compression + fine-grained selection, hardware-aligned, trained end-to-end. Claim: matches dense attention on general/long-context/instruction benchmarks while saving compute/bandwidth.

Expert sparsity: MoE continued scaling (V3 671B → V4 1.6T).

Knowledge sparsity: Engram (arXiv:2601.07372) — a scalable conditional memory module (N-gram lookup modernised) that sits alongside MoE. Under equal parameter/FLOP budgets, a 27B Engram model outperformed MoE baseline on MMLU (+3.4), CMMLU (+4.0), BBH (+5.0), etc., and lifted Multi-Query NIAH from 84.2 to 97.0. A U-shaped scaling law governs capacity allocation between MoE and memory.

NSA/DSA productised in DeepSeek-V3.2-Exp (Sept 2025) with quality parity (MMLU-Pro 85.0 both) and >50% API price drop; V3.2 (Dec 2025) merged DSA into mainline plus scalable RL framework and agentic data synthesis. Verification signal expansion: DeepSeek-GRM (arXiv:2504.02495) — generative pointwise reward modelling with SPCT training and parallel sampling/voting for non-verifiable tasks. DeepSeekMath-V2 (arXiv:2511.22570) shifted reward from final answer to a trained verifier of derivation quality, achieving IMO/CMO gold and Putnam 118/120 with extended test-time compute. Formal line: Prover-V1.5 (arXiv:2408.08152) RL with Lean feedback (miniF2F 63.5%); Prover-V2 (arXiv:2504.21801) recursive theorem decomposition + synthetic cold-start data (MiniF2F-test 88.9%, PutnamBench 49/658). Infrastructure: Insights into DeepSeek-V3 (arXiv:2505.09343) hardware feedback on memory/compute/interconnect constraints; DeepSeek-OCR (arXiv:2510.18234) renders text as images, decodes with visual tokens — 97% accuracy at 10× compression, 60% at 20×, beating GOT-OCR2.0 and MinerU2.0 in token efficiency.

Figure: Engram's U-shaped MoE/memory allocation vs validation loss (left); memory slots vs loss (right). Source: DeepSeek-AI, Conditional Memory via Scalable Lookup (arXiv:2601.07372), official repo figures/scaling_law.png .

Act V: From "How Long" to "Bytes per Token" (2026-01 to 2026-09)

DeepSeek-V4 preview (arXiv:2606.19348): V4-Pro 1.6T/49B, V4-Flash 284B/13B, 1M context, >32T pre-train tokens. Three architectural changes: hybrid attention (CSA+HCA), Manifold-Constrained Hyper-Connections, Muon optimizer. At 1M context, V4-Pro inference FLOPs/token = 27% of V3.2, KV cache = 10%. V4.1-Flash (arXiv:2609.19969): 552B backbone, decode 16B active / prefill 8B active; dual KV compression — cross-layer reuse (CSA2) + FP4 quantisation — yielding 890 bytes/token resident in HBM (≈1/4 of V4-Flash, 1/8 including SSD/host offload). Pre-train corpus: 45T multimodal tokens. DSec (arXiv:2609.22978) details the agentic sandbox platform: ~160 nodes/unit, 3M sandboxes/day, 380K+ concurrent, 5K+ creates/sec, stateful rollouts decoupled from preemptible GPU training, images loaded on-demand from 3FS. DeepSeek Harness (GitHub: https://github.com/deepseek-ai/deepseek-harness) — plugin-based agent framework (developer preview).

Figure: Scale trajectory — total params vs active params per token across generations.

Multimodal Side Lines: Two Approaches to Visual Granularity

Understanding path: DeepSeek-VL (arXiv:2403.05525) mixed web/PDF/OCR/chart data, hybrid encoder for 1024×1024; VL2 dynamic tiling encodes variable aspect ratios natively. Generation path: Janus decoupled encoders (understanding vs generation granularity differ), unified backbone; JanusFlow rectified flow in LM backbone; Janus-Pro (arXiv:2501.17811) scaled data/model only. No new multimodal papers after Jan 2025 in public record. Third approach: DeepSeek-OCR 2 (arXiv:2601.20552) semantic reordering of visual tokens instead of raster scan, further improving compression/accuracy trade-off.

Figure: OCR decode accuracy vs text/visual token ratio (a); OmniDocBench edit distance vs visual tokens/page (b). Source: DeepSeek-AI, DeepSeek-OCR (arXiv:2510.18234), official repo assets/fig1.png .

Putting the Three Cost Accounts Together

The three metrics have different baselines and visibility:

Training cost: first quantified at V2 (−42.5% vs 67B), then absolute at V3 (2.788M H800h), then silent.

KV cache: largest relative drops — V2 −93.3%, V4 10% of V3.2, V4.1-Flash 890 bytes/token (absolute).

Throughput: only explicit at V2 (×5.76), later folded into context/deployment narratives.

The author warns against direct cross-generation comparison (different baselines, context lengths, hardware). The internal trend: each generation shrinks the per-token denominator and reinvests the savings into context length (128K → 1M), post-training compute (RL, agentic synthesis), or model scale. Cache compresses faster than compute because it only changes what is stored, not what is computed.

Figure: Four cache/compute revisions — dense attention, MLA, sparse attention (DSA), CSA2+FP4. Source: community-drawn.

Where Reward Signals Come From

Post-training evolution is a continuous expansion of automatically verifiable reward :

Human reasoning traces (early) →

Verifiable outcomes (math/code) via GRPO (no value net) →

Generative reward modelling + inference-time voting (GRM) for non-verifiable tasks →

Self-verification: trained verifier judges derivation quality (Math-V2) →

Formal verification: Lean 4 as ground-truth verifier (Prover series) →

Interactive environments: synthetic agentic tasks, replayable sandboxes (DSec, V3.2 agentic pipeline).

All algorithmic changes (GRPO, pure RL, SPCT, verifier training) share one direction: replace human labour and extra models with more automatically judgeable samples. The trade-off: reported gains concentrate in math/code/formal reasoning — limited representativeness for general tasks.

Figure: Reward source migration — human traces → verifiable outcomes → reward models → self-verification → interactive environments. Source: community-drawn.

What the Experiments Do Not Prove

No cross-vendor same-protocol benchmarks. Claims like "matches GPT-5" or "exceeds GPT-5" are self-reported under self-defined eval settings; Math-V2's 118/120 uses extended test-time compute.

No reproduction artefacts. Data recipes, RL environments, eval scripts unpublished. V3/V4 disclose scale/hours/stability but not data composition. R1's verifiable-task pipeline remains external.

No failure statistics. Only final metrics published; DSec uniquely mentions reward hacking as a mitigation target, implying a gap between optimised and true objectives.

No cross-scale extrapolation. Engram's U-law validated to 27B; MoE expert-splitting evidence from 2B. No larger-scale confirmation.

Engineering tricks not disentangled from capability. Self-consistency, tree search, test-time scaling often conflated with base model quality (e.g., Prover 46.3% vs 52% = sampling count; Math-V2 118/120 requires extended compute).

Baseline drift. Reports habitually use previous self as baseline; "no regression" (V3.2-Exp MMLU-Pro 85.0→85.0) is a successful efficiency swap, not a capability jump.

Summary: Three Concurrent Engineering Tracks

The chronicle reveals three interlocking workstreams:

Architecture track — shrinks per-token cost: MLA (cache), MoE (compute), DSA/CSA2 (context), Engram (knowledge offloaded from compute).

Post-training track — expands verifiable task supply: GRPO, pure RL, generative RM, self-verification, agentic environments.

Infrastructure track — enables the other two at scale: custom clusters, FP8 stack, 3FS, sandbox platform.

They advance asynchronously (architecture leads 2024, post-training shifts 2025, infrastructure stretches to support agentic RL) and relay rather than replace each other. Sparse attention only pays off with matching kernels/cluster; RL scale only works with rollout environments that hold up. This explains the "off-topic" systems papers in the lineup.

Final takeaway: Long-context competition has shifted from "maximum supported length" to "bytes per token amortised." V4.1-Flash's 890 bytes and DSec's sandbox concurrency are engineering metrics of this new regime. Whether they translate to externally verifiable capability gains awaits independent evaluation. A further corollary: DeepSeek's technical choices are tightly coupled to their proprietary engineering stack (custom kernels, inference stack, sandbox platform). Portability depends on equivalent systems investment, not just paper clarity — hence the distinction between "mechanism readable" and "reproducible."