Skip to content
DeepSeek-V4-Flash on 64GB CPU: Why 96GB Xeon Can't Load It

DeepSeek-V4-Flash on 64GB CPU: Why 96GB Xeon Can't Load It

The article explains how llama-modchip runs 284B DeepSeek-V4-Flash on a 64GB RAM CPU-only machine via aggressive quantization (1.71 bits/weight), while a 96GB Xeon server fails due to tensor format mismatch and missing AVX-512. It breaks down inference latency for long contracts (~10 minutes for 20k tokens) and proposes three evaluation metrics: time to first token, key-field extraction accuracy, and total processing time.

DataFunTalk
DataFunTalk
DataFunTalk

Quantization Enables 284B Model on 64GB RAM

The open-source project llama-modchip demonstrates running DeepSeek-V4-Flash (284B total parameters, ~13B active parameters per token, MoE architecture) on a 64GB DDR5 Linux machine with an AMD Ryzen 9 9955HX CPU and no GPU. The project quantizes the original 76.87 GiB model to 56.56 GiB (average 1.71 bits/weight ), using mixed precision: routing experts at ~1.56 bits/weight, sensitive layers at higher precision, and attention projections in a custom Fractal-64 format. At 256K context, resident memory is 57.6–58.9 GiB with zero swap (project's self-reported telemetry, Hacker News discussion with 362 points, 153 comments as of Nov 9).

Figure 1: DeepSeek-V4 Flash overall structure: CSA/HCA mixed attention alternating with DeepSeekMoE per layer, topped by prediction head and MTP module
Figure 1: DeepSeek-V4 Flash overall structure: CSA/HCA mixed attention alternating with DeepSeekMoE per layer, topped by prediction head and MTP module

Why 96GB Xeon Fails to Load

A reader on Hacker News attempted the same quantized model on a Haswell-era Xeon server with 96GB DDR4 . Two failures occurred: (1) tensor shape mismatch — loader expected dimension 4, file provided 6 — revealing the custom GGUF tensor type unrecognized by standard llama.cpp, Ollama, or LM Studio without adaptation; (2) illegal instruction crash from the precompiled binary, because the CPU lacks AVX-512 . The project documentation explicitly states its custom GGUF uses special tensor types and requires AVX-512BW for the optimized kernels that process multiple low-bitwidth groups per instruction and reuse data in registers/cache. Thus, raw memory capacity is insufficient without matching tensor format, runtime version, and CPU instruction set.

Figure 2: llama-modchip real-time telemetry on 64GB DDR5 pure CPU: resident 57.61 GiB, zero swap, zero OOM
Figure 2: llama-modchip real-time telemetry on 64GB DDR5 pure CPU: resident 57.61 GiB, zero swap, zero OOM

Inference Speed and Long-Contract Latency

Reported generation throughput: 9.06–9.95 tokens/s (default, Top-6 routing), up to 10.74 tokens/s (speed mode with Top-5/Top-4 routing and skipped compute). Prefill benchmark pp512 shows ~44 tokens/s for 512 input tokens. For a 20k-token contract (including prompt), prefill at 40 tokens/s takes ~8 minutes 20 seconds ; generating 800 tokens at 9–10 tokens/s adds 80–89 seconds . Total ~10 minutes before first usable output, excluding PDF parsing, queueing, and review. This latency suits background extraction but not interactive Q&A, where users must distinguish initial submission wait from follow-up questions that may reuse prefix cache.

Figure 3: CSA core structure: token-level compressor reduces KV entries to 1/m, Lightning Indexer selects Top-k, concatenated with sliding window entries fed into shared KV multi-query attention
Figure 3: CSA core structure: token-level compressor reduces KV entries to 1/m, Lightning Indexer selects Top-k, concatenated with sliding window entries fed into shared KV multi-query attention
Figure 4: Single-token compute and cumulative KV cache vs sequence length: DeepSeek-V3.2 vs V4-Pro vs V4-Flash comparison
Figure 4: Single-token compute and cumulative KV cache vs sequence length: DeepSeek-V3.2 vs V4-Pro vs V4-Flash comparison

Accuracy Risks in Contract Extraction

The project labels its default mode "Lossless" based on next-token logit similarity between runtime settings on the same 56.56 GiB weights. However, no public data shows accuracy on long-contract extraction . Example: a contract states "payment within 30 days after acceptance" in the main body, but a supplementary agreement overrides to "60 days for cross-border deliveries". Missing the exception clause yields a plausible but misleading result. The article proposes a three-metric evaluation on anonymized contracts with human-labeled key fields (payment terms, auto-renewal, liability caps, exceptions) and page citations:

Time to first visible token (and time to first usable result if reasoning shown).

Key-field extraction accuracy — amount, term, condition, and source text must all match; critical omissions tracked separately.

End-to-end time per contract — parsing, waiting, rework included.

Comparing default vs. speed mode on the same weights isolates routing-cut impact; comparing against higher-precision weights isolates quantization impact. Consistent prompts, output formats, and scoring are essential, including tests for hallucinated clauses.

Conclusion

The 64GB experiment pushes a massive MoE model into commodity hardware range. Whether it handles real workloads depends on the three metrics above — starting with that contract where payment terms are split across main text and supplement.

References: [1] Hacker News discussion & 96GB Xeon reproduction feedback; [2] llama-modchip v2.4.0 README & self-test table; [3] DeepSeek official V4-Flash specifications; [4] llama-modchip DeepSeek V4 optimization & operator analysis.

Code example

[2]llama-modchip v2.4.0 README 与自测表
[3]DeepSeek 官方 V4-Flash 模型规格
[4]llama-modchip 的 DeepSeek V4 优化与算子分析
Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

quantizationMixture of ExpertsCPU inferenceAVX-512GGUFDeepSeek-V4-Flashcontract extractionllama-modchip
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.