Skip to content
CPU-Only 284B LLM at 9 Tokens/sec: Why Memory Capacity Now Trumps GPU

CPU-Only 284B LLM at 9 Tokens/sec: Why Memory Capacity Now Trumps GPU

The llama-modchip project demonstrates running a 284B parameter MoE model on a 64GB RAM CPU-only machine via aggressive quantization to 56.6GB and kernel optimizations, but achieves only 9 tokens/sec decoding speed, revealing that memory capacity—not GPU—is becoming the key bottleneck for local large language model inference, with practical implications for Mac memory sizing.

Lao Guo's Learning Space
Lao Guo's Learning Space
Lao Guo's Learning Space

The article analyzes the llama-modchip project, which enables running the 284B-parameter DeepSeek-V4-Flash MoE model on a standard 64GB RAM desktop with zero GPU. The author cross-references the official README, Hacker News discussions, and multiple independent benchmarks to separate hype from reality.

What llama-modchip Actually Is

llama-modchip is not a new model but a toolkit: ggml/llama.cpp kernel patches + quantization converter + runtime tools . It targets the core bottleneck of MoE models—memory capacity, not compute. DeepSeek-V4-Flash has 284.33B total parameters but activates only ~13.17B per token. The challenge is fitting the full weight set into RAM.

The project achieves this through three engineering steps:

Weight repacking to 56.56 GiB (1.71 bit/weight average) , down from the standard 76.87 GiB. Critical tensors (embeddings, 43 shared experts, attention keys) stay at Q4_K/Q5_K; the 256 rarely-used routed experts are compressed to ~1.56 bit.

CPU-cache-aware kernel patches using AVX-512BW register tiling, L1/L2/L3 cache-aware scheduling, and non-temporal memory accesses to prevent cold expert weights from evicting hot KV cache. On a 16-core Zen 5, prefill throughput rises from 25 to 44.36 tok/s.

Speculative decoding with a tiny 1.64 MB "Stockfish-PVS" drafter (named after the chess engine) that predicts tokens for the large model to verify in batch, combined with KV snapshots in /dev/shm to restore long contexts in 13–33 ms.

Speed Reality: Fitting ≠ Usable

Benchmarks on a 16-core Zen 5 (Ryzen 9 9955HX) show:

Prefill: 43.16–44.36 tok/s (77% improvement over previous version).

Pure decode: 9.06–9.95 tok/s (~550 characters/minute).

Speculative decode: 11.5–14.7 tok/s (the "fast" ceiling).

At 9 tok/s, a 10k-token prompt takes ~225 seconds (nearly 4 minutes) to produce the first token. The project itself admits it suits single-user, offline, few-minute-startup scenarios —not high-concurrency or interactive production use.

The author notes that many social posts conflate numbers from the 30B tier (which hits 25+ tok/s speculative) with the 284B tier, and the claimed 99.15% cosine similarity is self-reported, not independently reproduced.

The Practical Sweet Spot: 30B Models

For daily use, the 30B MoE tier is far more compelling. Nemotron-3.5-30B-A3B quantizes to 14.35 GiB, runs on 16–32GB laptops, and delivers 25 tok/s base decode, 60.4 tok/s speculative, up to 111.6 tok/s on 8-channel Threadripper. Other adapted models include Qwen3-30B-A3B, Qwen3-14B, and Nemotron-H-8B. This tier crosses the "usable" threshold for chat, code drafting, and lightweight agents.

Implications for Mac Users

Don't directly map the 64GB x86 numbers to Mac. All published benchmarks are on x86 (AVX-512/AVX2). Mac uses ARM64 NEON/SVE/Apple Metal; no unified, verifiable Mac benchmark exists yet.

Unified memory must serve the OS first. A 16GB Mac only has ~11–12GB for models. The "16GB runs 30B" claim is for PC; Mac needs at least 24GB to comfortably run the 14.35 GiB 30B package.

If you have 128GB+ unified memory, you don't need extreme compression. A 128GB Mac Studio can hold the 56.6GB weights with room to spare, allowing higher-quality 2-bit/3-bit quantization that preserves intelligence better than the 1.56-bit expert compression forced on 64GB x86 machines.

The project's real signal: the entry barrier for local giant MoE models is shifting from "do you have a high-end GPU?" to "do you have enough RAM?" For anyone deciding between 16GB, 32GB, or 128GB Mac configurations, this is the most direct reference point.

Closing Takeaways

llama-modchip is a remarkable engineering feat that brings a "rack-scale GPU" model to a commodity high-RAM PC. But "runs" and "usable" are separated by the 9 tok/s reality. The author's advice: don't treat this as a "CPU revolution" proof; treat it as a "memory matters more than VRAM" signal. When buying your next machine, shifting budget toward RAM capacity may pay off more than debating GPU models.

All speed and size figures come from the project README, Hacker News threads, and public media benchmarks as of 2026-11-16; versions differ, and the article flags each discrepancy. Quantization quality remains the murkiest part—community members have questioned the 76.87 GiB baseline; final quality should be judged by running the project's own eval commands.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

quantizationspeculative decodingMoElocal LLMCPU inferencememory capacityMac hardwarellama-modchip
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.