Local LLMs Viable: Sparse Attention, MoE, KV Compression, Multi‑Token Prediction

Model Landscape

Qwen 3.6 released a 27B dense open‑source model and a 35B mixture‑of‑experts (MoE) model that activates roughly 3B parameters per token. Google’s Gemma 4 appears in several sizes with strong performance. GLM‑5 is a 744B MoE model, and Kimi K2.6 reaches a trillion total parameters with 32B active per token, though both demand high memory. DeepSeek previewed V4 in April, offering Flash and Pro MoE variants that support a million‑token context. Most of these models avoid loading the entire parameter set into memory, activating only a small fraction per token.

Sparse Attention Mechanism

Standard attention scales quadratically with context length, causing the computational work to increase four‑fold when the context doubles and a hundred‑fold when it grows ten‑fold. This cost makes long contexts expensive and contributes to the historically small window sizes. DeepSeek pioneered sparse attention with a “lightning indexer” that runs in FP8 on a separate CUDA stream, scoring early tokens to select a top‑k subset for full‑resolution processing while keeping a small sliding window for local coherence. This reduces complexity from quadratic to roughly linear on the selected set, and DeepSeek reports that V4‑Pro’s per‑token FLOPs on a million‑token context are about one‑quarter of V3.2’s, with KV cache usage reduced to one‑tenth.

Mixture of Experts (MoE)

MoE enables trillion‑parameter models by routing each token to a few expert sub‑networks instead of a single dense feed‑forward layer. Kimi K2.6 uses 384 experts, activating eight plus a shared expert per token. GLM‑5 activates about 40B of its 744B parameters per token. While MoE saves compute and bandwidth, all experts must reside in memory, making the overall memory footprint heavy. This tension makes unified‑memory systems (e.g., Apple Mac Studio, AMD Strix Halo) especially suitable for MoE models.

KV Cache Issue

For long‑context inference, the dominant memory cost is the key‑value (KV) cache, which grows linearly with context length and must stay in high‑speed memory. Two main mitigation strategies emerged in 2026: (1) Multi‑head latent attention compresses the KV cache into a low‑rank latent representation, cutting space by roughly 90 % (adopted by DeepSeek and variants in Kimi); (2) Storing the cache at lower precision (FP8 or FP4) halves or quarters memory usage with minimal accuracy loss that can be recovered through training. Combining compressed attention with quantized cache pushes the memory bottleneck far back.

Multi‑Token Prediction

Traditional generation emits one token at a time, limiting throughput by memory bandwidth. Multi‑token prediction, first validated at scale by DeepSeek‑V3, uses a cheap “draft model” to hypothesize several tokens, which the full model then verifies in parallel, accepting matches and discarding the rest. DeepSeek reports an 85‑90 % acceptance rate for the second token, yielding about 1.8× higher throughput. Gemma 4 incorporates a small draft model sharing embeddings and KV cache with the main model, adding negligible cost. The approach is lossless—final outputs match those of the full model—though benefits vary with task difficulty and can waste work on rejected drafts.

4‑Bit Quantization

Four‑bit (FP4) precision, appearing as NVFP4 and MXFP4, has moved from research to deployment. OpenAI released MXFP4‑based gpt‑oss, and Nvidia’s Blackwell hardware natively supports FP4. A Qwen 3.6 27B model quantized to near‑4‑bit occupies ~17 GB, dropping to ~14 GB with NVFP4; quantization‑aware training recovers most lost accuracy. FP4 may degrade small or sensitive models due to block‑size and outlier handling, but for larger models it becomes a reasonable default rather than a compromise.

Memory Supply Crisis

Despite engineering gains, 2026 saw a sharp rise in hardware prices as AI competitions drove massive demand. DRAM prices jumped 90‑98 % YoY, with PC‑grade DRAM and NAND SSDs roughly doubling. HBM production shifted toward data‑center use, reducing availability for consumer‑grade memory. SK Hynix indicated capacity sold out for the coming year, with relief not expected until late 2027. Consequently, running local models on a single GPU stack becomes less feasible, prompting interest in unified‑memory systems (e.g., Mac Studio, AMD Strix Halo) that combine CPU and GPU memory, which suit MoE models that need large capacity but modest bandwidth per inference.

Gap with Closed‑Source Models

Epoch’s analysis shows top open‑weight models lag closed‑source front‑runs by about four months, slightly wider than the three‑month average previously observed. In programming and agent tasks, the gap narrows to a few points, often imperceptible, while high‑difficulty reasoning and novel mathematics still favor closed models. Artificial Analysis’s June index rates Kimi K2.6 at 54 versus GPT‑5.5’s 60 and Claude Opus 4.7’s 57; DeepSeek V4 Pro matches Sonnet 4.6, whereas smaller dense models like Qwen 27B and Gemma 31B fall a tier behind. Benchmarks are increasingly optimized for score, inflating public numbers, and Claude Fable 5 remains untracked.

Why Run Models Locally

Experimenting with local inference reveals performance differences—e.g., 14 tokens/s on one machine versus 40 tokens/s on another—and exposes KV cache memory consumption as context grows. Open‑source stacks let users pull models on release day, apply quantization and compression to fit any idle hardware, fine‑tune with private data, and keep sensitive information on‑premises. All discussed advances—sparse attention, MoE routing, latent KV compression, multi‑token prediction, and 4‑bit quantization—are publicly documented and merged in code repositories, preserving openness and offering users genuine choice.