284B DeepSeek on 64GB CPU: Quantization Works, 96GB Xeon Fails on AVX-512
The llama-modchip project runs a 284B-parameter DeepSeek-V4-Flash model on a 64GB RAM CPU-only machine via aggressive quantization to 1.71 bits/weight (56.56 GiB), achieving 9–10 tokens/sec, but a 96GB Xeon server fails due to custom GGUF tensor types and missing AVX-512, highlighting hardware-software compatibility gaps.
01 Quantization Shrinks Model by 26% to Fit 64GB
DeepSeek-V4-Flash officially lists 284B total parameters and ~13B active parameters per token using a Mixture-of-Experts (MoE) architecture. The llama-modchip project further compresses a 76.87 GiB quantized file down to 56.56 GiB (≈26% reduction), reaching an average of 1.71 bits/weight . Routing experts are quantized to ~1.56 bits/weight , while sensitive layers retain higher precision; attention projections use a custom Fractal-64 format. The same compressed weights are used for both default (Top-6 routing) and speed modes (Top-5/4 with skipped compute). Process resident memory at 256K context is reported at 57.6–58.9 GiB with zero swap usage.
02 Why 96GB Xeon Fails to Load
A Hacker News user tested on a Haswell-era Xeon with 96GB DDR4. Two failure modes appeared:
Tensor shape mismatch : loader expected dimension 4, file provided 6 — revealing the custom GGUF tensor types that unadapted llama.cpp, Ollama, or LM Studio cannot recognize.
Illegal instruction : the pre-compiled binary required AVX-512 , absent on the older CPU.
Thus, before memory capacity matters, the weight shards, runtime version, and CPU instruction set must align. The extra 32GB did not help these compatibility errors. At decode time, each token generation pulls weights from memory, dequantizes, and runs matrix ops; Zen 5 leverages AVX-512BW to process multiple low-bit groups and reuse data in registers/cache. Memory capacity sets how much data stays resident, but bandwidth, cache, and compute cores jointly determine token throughput.
03 Latency Breakdown: 10 Tokens/sec ≠ Interactive Speed
README highlights 256K context and ~10 tokens/sec generation (default: 9.06–9.95 tok/s; speed mode up to 10.74 tok/s). The 44 tok/s figure comes from a pp512 prefill benchmark (512 input tokens). For a 20k-token contract (including prompt) at an assumed 40 tok/s prefill, input processing alone takes ~8 min 20 sec ; generating 800 tokens at 9–10 tok/s adds 80–89 sec , totaling nearly 10 minutes before first usable output — excluding PDF parsing, queueing, and review. This explains why the setup suits background extraction but struggles for interactive Q&A , where users must distinguish initial submit latency from follow-up turns that may reuse prefix cache.
04 Accuracy Risks: Missing a Clause Is Costlier Than Latency
Example: contract states "payment within 30 days after acceptance", but an addendum overrides "cross-border batches: 60 days". The model may extract the 30-day term and cite the main text page, yet miss the addendum condition — producing a plausible but misleading JSON. The project labels default mode "Lossless" based on next-token scoring similarity across runtime settings, but no public data shows quantization-level accuracy on long contract extraction .
Recommended evaluation:
Sample anonymized contracts; human-label payment terms, auto-renewal, liability caps, exception conditions with page references.
Compare default vs. speed mode on same weights to isolate routing-cut impact.
If possible, run higher-precision weights of same version to separate quantization effect from runtime trimming.
Keep prompts, output format, and scoring consistent; include absent clauses to test hallucination.
Three metrics to track:
Time to first visible token (and time to first usable result if reasoning shown).
Key-field extraction accuracy : amount, term, condition, and source evidence must all match; critical exception misses counted separately.
End-to-end human confirmation time : parsing, waiting, rework included. If speed mode saves seconds but forces reviewers to hunt missing addenda, net gain appears only in this final measure.
The 64GB experiment pushes a formerly single-machine-infeasible model into commodity hardware range. Whether it handles real workloads can be tested on that payment-term-scattered contract: when it returns "who pays what, under which conditions, within how many days" with citations, then measure total processing time.
References
[1] Hacker News discussion and 96GB Xeon reproduction feedback
[2] llama-modchip v2.4.0 README and self-test tables
[3] DeepSeek official V4-Flash model specifications
[4] llama-modchip DeepSeek V4 optimization and operator analysis
Code example
[2]llama-modchip v2.4.0 README 与自测表
[3]DeepSeek 官方 V4-Flash 模型规格
[4]llama-modchip 的 DeepSeek V4 优化与算子分析Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
