Skip to content
Why Read the Whole Network? NVIDIA SEED Cuts Prefill Compute, Boosts Throughput 42%

Why Read the Whole Network? NVIDIA SEED Cuts Prefill Compute, Boosts Throughput 42%

NVIDIA's SEED technique exploits read-write asymmetry in hybrid models, using shallow layers to write deep-layer memories and compressing recurrent states via low-rank factorization, achieving 16-42% higher per-GPU throughput on Qwen3.8-Flash-Next with minimal quality loss across five models.

Machine Heart
Machine Heart
Machine Heart

Overview

NVIDIA's SEED (Shallow Encoding Emits Deep Memories) technique accelerates inference for hybrid architectures (combining softmax attention and linear/state-space layers) by exploiting two structural asymmetries: a read-write compute gap during prefill and a low-rank structure in recurrent states during decoding.

Shallow Prefill via Write-Read Asymmetry

In Qwen3.8-Flash-Next (48 layers), only the last prefill token needs the full network to predict the first response token. Earlier tokens only need to write KV cache and recurrent states for deeper layers. SEED introduces emitters at an anchor depth k=31 that take shallow hidden states and directly emit the writes for layers 32–48, skipping the remaining attention and feed-forward computation.

The anchor depth is chosen by measuring the KL divergence between the emitter-produced writes and the original deep-layer writes, alongside an exact-copy test (retrieving identifiers from context). For Qwen3.8-Flash-Next, the untrained shallow-write KL is ~141× smaller than the early-exit KL for next-token prediction, confirming that writing memory requires far less depth than reading it. GLM-5.3-Flash required moving the anchor from k=27 to k=31 to raise exact-copy recovery from 269 to 530 of 595 identifiers, showing average loss can mask retrieval failures.

Emitters are initialized from original layer weights and distilled on fixed-text then teacher-generated responses. After distillation, per-token parameter FLOPs for non-last prefill tokens drop from 12.1 GFLOP to 9.1 GFLOP (~25% reduction). Generation still runs the full 48 layers.

Recurrent State Compression via Fixed-Direction Sink + Low-Rank Factors

Each of the 1,728 Gated DeltaNet heads maintains a 128×128 state matrix. SVD reveals a dominant singular direction accounting for ~85% median energy, with near-fixed orientation across unrelated inputs. The authors define a state sink as the projection onto the average value direction per head (median ~75% energy). Replacing the sink with another input's average sink increases loss by only ~0.002 nats/token (one-time intervention), but retaining only the sink at every decoding step adds ~1.66 nats/token (continuous intervention), proving the residual working memory is essential.

SEED decomposes each head's state into:

A shared fixed direction v̄ (same for all requests)

A per-request sink coefficient a Low-rank factors U (128×R) and V (128×R) for the working memory (R=16)

The model computes outputs directly from these factors without reconstructing the full matrix. Storage per head drops from 16,384 elements to 4,224 (~4× compression) at the same precision.

Crucially, the Gated DeltaNet, Kimi Delta Attention, and Mamba-2 update rules preserve this factorized form exactly between compressions; only the periodic rank-R truncation introduces approximation. With compression period W=16, factors grow by one column per token and are recompressed every 16 tokens. Post-compression state size for Qwen3.8-Flash-Next falls from 113.2 MB to 29.2 MB; cycle average is 43.4 MB. Across five models, post-compression ratios range 2.0–3.9×, cycle averages 1.3–2.6×. Attention KV cache remains uncompressed.

Error analysis under open-loop conditions (same inputs, contractive decay) shows error bounds independent of compression steps. In free generation up to 16K steps, distribution shift from state compression grows only 1.2–1.4× from the first 4K to the last 4K steps.

Quality Evaluation

Five hybrid models (Qwen3.8-Flash-Next, Nemotron-3.5-Lightning, Kimi-Linear, GLM-5.3-Flash, Nemotron-3.5-Super) across six checkpoints and 11 benchmarks. Each question paired with original model using identical sampling seeds. Of 66 checkpoint×benchmark combinations, 63 scores fall within two standard errors of the original or higher. Anchor depth and rank selected via probing criteria (distribution shift + exact copy), not exam scores.

End-to-End Serving Gains on GB300 + SGLang

Replay of InferenceX coding-agent sessions (900 s per concurrency point). Both original and SEED use FP32 recurrent states; measurements include emitter and periodic compression overhead.

Aggregated NVFP4 (TP1): Peak throughput 80.2k → 96.0k token/s/GPU (+19.7%).

BF16 prefill-decode disaggregated (1P:1D, 2 GPUs each): Peak 33.9k → 48.1k token/s/GPU (+42%).

P90 user-facing latency matched: Throughput +23% to +42% at paper's selected operating points.

In aggregated deployment, smaller states raise prefix-cache hit rate from ~55% to ~85% at 40 concurrent sessions (NVFP4 TP1), reducing recomputation. Shallow prefill further cuts miss cost. In disaggregated deployment, the prefill instance is the bottleneck; shallow prefill alone gives ~35% peak gain, state compression alone ~5%, combined ~42%.

Limitations and Future Work

SEED does not compress attention KV cache, so its memory benefit diminishes when KV cache dominates (long contexts). Prompt-code prefix storage is discussed as a future direction and not included in reported throughput. Code, new components, and training data are slated for release.

Paper: Shallow Encoding Emits Deep Memories (arXiv:2611.06084v1) - https://arxiv.org/abs/2611.06084v1. Authors: Mingyuan Ma, Yangmin Li (co-lead), Harris Nover, Zheyu Fu, Po-Han Huang, Alex Settle, Ashwin Nanjappa (all NVIDIA). Project page: https://latent-voyager-seed.static.hf.space/index.html.
Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

inference optimizationNVIDIAlow-rank factorizationhybrid modelsSEEDQwen3.8-Flash-Nextprefill accelerationstate compression
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.