Skip to content
284B DeepSeek on 64GB CPU: Quantization Works, 96GB Xeon Fails on AVX-512

284B DeepSeek on 64GB CPU: Quantization Works, 96GB Xeon Fails on AVX-512

The llama-modchip project runs a 284B-parameter DeepSeek-V4-Flash model on a 64GB RAM CPU-only machine via aggressive quantization to 1.71 bits/weight (56.56 GiB), achieving 9–10 tokens/sec, but a 96GB Xeon server fails due to custom GGUF tensor types and missing AVX-512, highlighting hardware-software compatibility gaps.

DataFunSummit
DataFunSummit
DataFunSummit

01 Quantization Shrinks Model by 26% to Fit 64GB

DeepSeek-V4-Flash officially lists 284B total parameters and ~13B active parameters per token using a Mixture-of-Experts (MoE) architecture. The llama-modchip project further compresses a 76.87 GiB quantized file down to 56.56 GiB (≈26% reduction), reaching an average of 1.71 bits/weight . Routing experts are quantized to ~1.56 bits/weight , while sensitive layers retain higher precision; attention projections use a custom Fractal-64 format. The same compressed weights are used for both default (Top-6 routing) and speed modes (Top-5/4 with skipped compute). Process resident memory at 256K context is reported at 57.6–58.9 GiB with zero swap usage.

DeepSeek-V4 Flash architecture: CSA/HCA mixed attention with DeepSeekMoE alternating per layer, topped by prediction head and MTP module
DeepSeek-V4 Flash architecture: CSA/HCA mixed attention with DeepSeekMoE alternating per layer, topped by prediction head and MTP module
llama-modchip telemetry on 64GB DDR5 CPU-only machine: resident 57.61 GiB, zero swap, zero OOM
llama-modchip telemetry on 64GB DDR5 CPU-only machine: resident 57.61 GiB, zero swap, zero OOM

02 Why 96GB Xeon Fails to Load

A Hacker News user tested on a Haswell-era Xeon with 96GB DDR4. Two failure modes appeared:

Tensor shape mismatch : loader expected dimension 4, file provided 6 — revealing the custom GGUF tensor types that unadapted llama.cpp, Ollama, or LM Studio cannot recognize.

Illegal instruction : the pre-compiled binary required AVX-512 , absent on the older CPU.

Thus, before memory capacity matters, the weight shards, runtime version, and CPU instruction set must align. The extra 32GB did not help these compatibility errors. At decode time, each token generation pulls weights from memory, dequantizes, and runs matrix ops; Zen 5 leverages AVX-512BW to process multiple low-bit groups and reuse data in registers/cache. Memory capacity sets how much data stays resident, but bandwidth, cache, and compute cores jointly determine token throughput.

03 Latency Breakdown: 10 Tokens/sec ≠ Interactive Speed

README highlights 256K context and ~10 tokens/sec generation (default: 9.06–9.95 tok/s; speed mode up to 10.74 tok/s). The 44 tok/s figure comes from a pp512 prefill benchmark (512 input tokens). For a 20k-token contract (including prompt) at an assumed 40 tok/s prefill, input processing alone takes ~8 min 20 sec ; generating 800 tokens at 9–10 tok/s adds 80–89 sec , totaling nearly 10 minutes before first usable output — excluding PDF parsing, queueing, and review. This explains why the setup suits background extraction but struggles for interactive Q&A , where users must distinguish initial submit latency from follow-up turns that may reuse prefix cache.

CSA core structure: token-level compressor reduces KV entries to 1/m, Lightning Indexer selects Top-k, concatenated with sliding window entries into shared KV multi-query attention
CSA core structure: token-level compressor reduces KV entries to 1/m, Lightning Indexer selects Top-k, concatenated with sliding window entries into shared KV multi-query attention
Single-token compute and cumulative KV cache vs sequence length: DeepSeek-V3.2 vs V4-Pro vs V4-Flash comparison
Single-token compute and cumulative KV cache vs sequence length: DeepSeek-V3.2 vs V4-Pro vs V4-Flash comparison

04 Accuracy Risks: Missing a Clause Is Costlier Than Latency

Example: contract states "payment within 30 days after acceptance", but an addendum overrides "cross-border batches: 60 days". The model may extract the 30-day term and cite the main text page, yet miss the addendum condition — producing a plausible but misleading JSON. The project labels default mode "Lossless" based on next-token scoring similarity across runtime settings, but no public data shows quantization-level accuracy on long contract extraction .

Recommended evaluation:

Sample anonymized contracts; human-label payment terms, auto-renewal, liability caps, exception conditions with page references.

Compare default vs. speed mode on same weights to isolate routing-cut impact.

If possible, run higher-precision weights of same version to separate quantization effect from runtime trimming.

Keep prompts, output format, and scoring consistent; include absent clauses to test hallucination.

Three metrics to track:

Time to first visible token (and time to first usable result if reasoning shown).

Key-field extraction accuracy : amount, term, condition, and source evidence must all match; critical exception misses counted separately.

End-to-end human confirmation time : parsing, waiting, rework included. If speed mode saves seconds but forces reviewers to hunt missing addenda, net gain appears only in this final measure.

The 64GB experiment pushes a formerly single-machine-infeasible model into commodity hardware range. Whether it handles real workloads can be tested on that payment-term-scattered contract: when it returns "who pays what, under which conditions, within how many days" with citations, then measure total processing time.

References

[1] Hacker News discussion and 96GB Xeon reproduction feedback

[2] llama-modchip v2.4.0 README and self-test tables

[3] DeepSeek official V4-Flash model specifications

[4] llama-modchip DeepSeek V4 optimization and operator analysis

Code example

[2]llama-modchip v2.4.0 README 与自测表
[3]DeepSeek 官方 V4-Flash 模型规格
[4]llama-modchip 的 DeepSeek V4 优化与算子分析
Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

quantizationMoECPU inferenceAVX-512GGUFDeepSeek-V4-Flashcontract extractionllama-modchip
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.