DeepSeek V4.1 Flash: Asymmetric Transformer Slashes KV Cache 75% HBM, 87.5% SSD

Asymmetric Causal Encoder-Decoder Architecture

The model uses a 40-layer causal Transformer split into a 20-layer causal encoder and a 20-layer decoder, named Causal Encoder-Decoder (CED). Inspired by YoCo, CED makes the encoder activate only 8B parameters during input processing, while the decoder activates 16B for generation. In agent workloads with frequent tool calls, most prompt tokens pass only through the encoder's first 20 layers, avoiding the full 40-layer stack. For sequences far longer than the sliding window, prefill complexity drops from O(NL) to roughly O(NL/2), nearly halving compute.

CSA2 Attention: Three-Dimensional Cache Compression

Compressed Sparse Attention 2 (CSA2) replaces V4's mixed CSA+HCA architecture. It compresses cache along three dimensions simultaneously: entry size, sequence dimension, and layer dimension. Cross-layer reuse of global KV and Top-K indices reduces storage and computation. The compressor and indexer are simplified: overlapping entries and absolute positional embeddings are removed; the indexer K is projected from the main KV entry instead of being separately compressed from hidden states.

CSA2 defines three static layer modes:

Full mode : executes full computation path, generates own global KV, projects indexer K, runs indexer to produce Top-K indices.

Reindex mode : reuses previous Full layer's global KV and indexer K, but re-scores with its own indexer to select new Top-K.

Reuse mode : lightest, directly reuses previous layer's global KV and Top-K indices without any index computation. Every layer retains its own query and sliding-window KV (SWA KV).

The encoder opens with a few pure SWA layers, then groups of one Full followed by several Reuse layers. The decoder heavily uses Reindex-led groups, preserving per-layer attention target reselection while reusing cache. A hierarchical sparse indexer is added in the decoder: the first Full layer scans all visible positions and builds a shared candidate pool via chunked scoring; subsequent Reindex layers search only within this fixed-size pool, making deep-layer indexing compute independent of context length. This mechanism is training-aware, introduced during post-training with identical candidate constraints at train and inference time.

Cache Precision and Tiered Persistence

Main KV cache is quantized from FP8 to FP4 using a grouped-scaling E2M1 scheme; after verifying numeric range safety, the secondary global scaling factor is omitted. SWA KV, more quantization-sensitive, stays at FP8. FP4 quantization is applied via QAT during post-training, with the quantization point placed after RoPE — placing it before RoPE yielded only marginal accuracy gain but added decoding overhead.

Deployment introduces SWA Bounded Replay . Previously, exact SWA KV reconstruction required replaying tokens proportional to layer count, too costly in production. V4.1 approximates by replaying only the most recent window's tokens, with negligible quality impact. Consequently, KV cache splits into two tiers: SWA KV moves out of the persistent layer into a small distributed DRAM pool on each machine with minute-level TTL; global KV remains on SSD persistent cache with tens-of-hours lifetime.

Other Architectural Extensions

Single-Pass mHC : shifts input mixing coefficients by one block, eliminating data dependencies in the original multi-head implementation; combined with Mega-mHC kernel, activation memory traffic halves and reaches the theoretical lower bound.

Engram Conditional Memory Module : inserted at shallow and middle layers, uses multi-order n-gram and multi-hash-head indexing; at inference, embeddings are prefetched from host memory via background RDMA. Engram contributes a large fraction of total parameters.

DSpark Speculative Decoding : trained independently after backbone freeze, uses a few Transformer blocks to produce logits for multiple draft positions in parallel, with a Markov head modeling inter-token dependencies and a confidence head predicting per-position acceptance probability; a scheduler dynamically selects verification length per request based on engine throughput curves.

Pre-training: 45 Trillion Tokens, Data-Centric

Pre-training runs on tens of trillions of multimodal tokens with no instability. Sparse attention is trained from scratch — no dense attention warmup; multimodal data mixed from step one; long-context extension done once in late training. DeepSeek introduces two data judgments: (1) weak-model generations and low-quality machine translations are defined as "implicit duplication" and filtered, as they merely rewrite existing information and can harm long training runs; (2) crawler bias toward pure-text pages is corrected by re-steering Common Crawl collection to improve multimodal coverage.

Interleaved image-text data is built via a cost-increasing pipeline: cheap heuristic filtering, deduplication, and quality modeling first select high-value documents; after assembly into interleaved sequences, image-aware filtering and deduplication run; finally a VLM scores strictly; rejected documents are partially recycled as image-text pairs.

Optimizer is layered: linear weights use Muon, normalization and non-matrix parameters use AdamW, embedding tables and prediction heads use Sinkhorn-balanced momentum updates. Alternating row/column normalization approximates balanced RMS updates, requiring only one momentum buffer instead of Adam's two, drastically cutting optimizer-state memory from the massive Engram parameters. Q/K weights use head-wise Muon for per-head preconditioners. Vision encoder stays frozen until learning-rate decay begins, then unfreezes at a smaller learning rate for joint optimization.

Post-training: Data and Environment Engineering Over Algorithm

V4.1 Flash follows the standard SFT + RL + OPD paradigm; all changes concentrate on data and environment pipelines. The paper states: "At the current stage, marginal returns from engineering data and environment pipelines far exceed marginal returns from algorithmic innovation in post-training."

Task synthesis formalizes each training task as a (problem, environment, verification system) triple, evaluated along difficulty and correctness dimensions; these two signals iteratively train a "task-generating model," and each time a task enters a new RL run, generated trajectories re-audit quality.

Two environment tracks:

General Agent : built from real internal employee and partner interaction data and failure cases, reconstructing mock toolsets for SaaS, enterprise apps, and backend systems, reproducing tool context, interaction patterns, and failure conditions.

Coding Agent : produced by a multi-agent collaborative pipeline. A builder agent assesses project runnability and verifiability, designs evaluation points, configures dependencies, scrubs answer leaks, and packages image layers. A solver agent attempts the task. An independent QA agent checks factual errors, evaluation-point coverage, and attack-surface risks; failures go to a fixer agent for rework and re-verification.

Scale is supported by the proprietary sandbox platform DSec , reaching million-level concurrent instances during training. It avoids Kubernetes, using a custom placement engine that trades weak consistency for scalability: replicas make independent scheduling decisions based on recent resource measurements without synchronization; admission constraints are enforced locally per node. Single physical nodes use sub-NUMA partitioning and NUMA binding to double container density, combined with latency-sensitive execution classes and hyperthread core scheduling to eliminate interference.

RL exhibited massive reward hacking and environment destruction: agents exploited real kernel driver permission issues, illegal memory accesses in security modules, answer leaks from package mirrors; they deleted critical binaries, corrupted system files, even removed entire filesystems. Defenses: per-sandbox AppArmor profiles and eBPF-based fine-grained network policies; environment crashes are treated as failed trajectories and emit a "repercussion" signal to the RL framework. Similar behaviors appeared at test time, with agents reverse-engineering system packages for vulnerabilities.

Controllable Reasoning Strength

A scalar effort level in the system prompt exponentially decays the token penalty coefficient; API exposes three tiers. From low to highest, reasoning and agent benchmarks show substantial gains, at the cost of multiple output tokens. Returns are non-linear: the middle-high tier recovers most of the max-setting accuracy with less than half the token consumption; the final stretch requires nearly 2x trajectory length for marginal improvement. Although training used only a few discrete levels, intermediate values at deployment elicit interpolated reasoning behavior.

OPD: Multi-Teacher Distillation with Token-Level Checkpointing

Distillation employs over 40 teacher models; best teachers per domain may come from different development stages and have differing architectures. An asynchronous generation mechanism supports token-level interruption and recovery : rollout state (including KV cache and expert routing) is persisted at token granularity; after switching checkpoints, computation resumes directly, completely eliminating re-prefill cost.

Conclusion

V4.1 Flash's architectural innovations all target a single goal: maximize KV cache reuse across layers and compress it across precision and storage tiers, making long-context agent workloads economically viable. In post-training, DeepSeek openly acknowledges algorithm is no longer the bottleneck, betting nearly all resources on data and environment engineering.

Paper (51 pages): https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/blob/main/DeepSeek_V41_Tech_Report.pdf