DeepSeek V4 Flash: Open‑Source Release, Extreme Quantization Cuts Local Deployment Cost

DeepSeek V4‑Flash Official Release

The DeepSeek V4‑Flash‑0731 model has been released and open‑sourced. Using the same architecture and parameter count as the preview, the team performed post‑training to teach the model how to answer questions, invoke tools, and complete tasks, resulting in benchmark scores that surpass the V4‑Pro preview from three months earlier.

Performance vs. Cost

Artificial Analysis plots model intelligent score (vertical) against task‑completion cost (horizontal). The ideal model sits in the upper‑left corner—high intelligence, low cost. DeepSeek‑V4‑Flash‑0731 lies near the so‑called “DeepSeek slashing line,” indicating a strong balance of capability and affordability.

Local Deployment Considerations

For enterprise use, local deployment is often mandatory. Competing models such as GLM‑5.2 and Kimi‑K3 have far larger parameter counts and prohibitive deployment costs, whereas DeepSeek‑V4‑Flash’s 8‑bit native precision yields a 167 GB model file (304 B parameters).

vLLM Deployment

The official local deployment recommends vLLM, requiring version 0.25 or newer and NVIDIA Hopper or Blackwell GPUs. The vLLM recipe also includes an H200 single‑node PD‑split configuration (4 cards prefill + 4 cards decode) for inference clusters.

Quantization Details

DeepSeek‑V4‑Flash applies Quantization‑Aware Training (QAT). In the official checkpoint, routed experts occupy 96 % of the model and are stored in MXFP4 format; the remaining 4 % use FP8 or BF16. Unsloth’s GGUF conversion performs a bit‑wise repack without altering expert weights, preserving exact bits while converting FP8 back to BF16 with zero rounding.

A counter‑intuitive finding is that not all 4‑bit quantizations are equal. Unsloth’s UD‑Q4_K_XL keeps expert weights at full precision, whereas some third‑party converters re‑quantize experts to Q4_K, resulting in a larger file (164.6 GB) and worse quality (KLD 0.029).

Benchmark Comparison

Key numbers from the benchmark table:

Official checkpoint: 156.4 GB, PPL 4.5319, KL 0, top‑token 100 %.

UD‑Q8_K_XL: 161.9 GB, PPL 4.5319, KL ≈0 (lossless), top‑token 100 %.

UD‑Q4_K_XL: 155.1 GB, PPL 4.5335, KL 0.0102, top‑token 96.28 %.

bartowski MXFP4: 156.0 GB, PPL 4.5351, KL 0.0105, top‑token 96.18 %.

antirez Q4KExperts‑F16: 164.6 GB, PPL 4.5726, KL 0.0290, top‑token 93.94 %.

antirez IQ2XXS: 86.7 GB, PPL 6.1518, KL 0.4207, top‑token 77.92 %.

UD‑Q8_K_XL matches the original checkpoint’s perplexity to four decimal places, a rarity in quantization.

Choosing a Quantization Level

Based on available memory (total RAM + VRAM):

> 169 GB → use UD‑Q8_K_XL (lossless).

≈ 162 GB → use UD‑Q4_K_XL (quality loss negligible, KL 0.0102).

128 GB → use UD‑IQ3_XXS as a fallback.

< 128 GB → recommend using an API instead of local deployment.

The official memory thresholds (including KV cache and context overhead) are 92 GB (1‑bit), 102 GB (2‑bit), 110‑135 GB (3‑bit), 162 GB (4‑bit, near‑lossless), and 169 GB (Q8_K_XL, lossless).

Running the Model

Compile llama.cpp with CUDA disabled for CPU‑only runs and Apple Metal disabled:

apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON
cmake --build llama.cpp/build --config Release -j --clean-first \
    --target llama-cli llama-mtmd-cli llama-server llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp

The simplest way to run is similar to ollama run:

export LLAMA_CACHE="unsloth/DeepSeek-V4-Flash-0731-GGUF"
./llama.cpp/llama-cli \
    -hf unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-Q8_K_XL \
    --temp 1.0 \
    --top-p 1.0 \
    --min-p 0.0

Because the model files exceed 100 GB, it is advisable to download manually and filter by desired quantization level:

hf download unsloth/DeepSeek-V4-Flash-0731-GGUF \
    --local-dir unsloth/DeepSeek-V4-Flash-0731-GGUF \
    --include "*UD-Q8_K_XL*"

Three ways to control the reasoning effort are provided via --chat-template-kwargs JSON strings, and llama.cpp now supports --reasoning on/off. Windows PowerShell users must escape quotes accordingly.

Template Note

DeepSeek does not ship a Jinja‑format chat template; only an encoding folder is provided. Unsloth supplies a compatible Jinja template that aligns over 4 000 dialogue rounds with the official baseline and restores the missing reasoning_content field.

Before using third‑party GGUF files, verify that the template and MXFP4 repack have been handled; otherwise the model may run but produce incorrect outputs.

Conclusion

DeepSeek’s V4‑Flash demonstrates that post‑training can substantially boost capability without increasing model size, and that extreme quantization can dramatically lower local deployment costs. By open‑sourcing the model and its tooling, DeepSeek enables enterprises to adopt a high‑performing, cost‑effective LLM that can be run on‑premises and remain under their control.