Unsloth’s Dynamic NVFP4 Makes Qwen3.6 Run 2.5× Faster Than NVIDIA’s Official Quantization

Conclusion

Qwen3.6‑27B runs in 24 GB VRAM on Blackwell GPUs and is 2.5× faster than NVIDIA’s standard NVFP4.

Qwen3.6‑35B‑A3B reaches 17,561 tokens/s under high concurrency on a B200.

Accuracy on MMLU‑Pro, GPQA and AIME 2025 matches BF16.

FP8 KV‑Cache calibration doubles the usable context length.

Qwen3.6 Overview

Qwen3.6‑27B : 270 B dense parameters, flagship coding ability (SWE‑bench Verified 77.2).

Qwen3.6‑35B‑A3B : 350 B total parameters with a 3 B active‑parameter MoE, delivering high inference efficiency.

Key capabilities include 256 K native context (extendable to 1 M with YaRN), support for 201 languages, strong agentic coding performance, thinking‑preservation for multi‑turn dialogue, and multimodal image/video understanding.

What Dynamic NVFP4 Does

Core Difference: W4A4 vs W4A16

NVIDIA’s official NVFP4 uses W4A16 (4‑bit weights, 16‑bit activations). The weights must be de‑quantized to 16‑bit before matrix multiplication, creating a bottleneck.

Unsloth’s approach is W4A4 : both weights and activations are 4‑bit, allowing native computation on the Blackwell FP4 Tensor Core and eliminating the de‑quantization step, which dramatically increases throughput.

Dynamic Layer Quantization

Important layers retain higher precision (up‑sampled) to preserve output quality.

Calibration dataset combines Unsloth’s own data with UltraChat, targeting programming, tool‑calling, and dialogue scenarios.

Chat template improvements fix encoding and tool‑calling consistency, reducing loops and known issues.

Built‑in MTP Acceleration

Multi‑Token Prediction (MTP) is Qwen3.6’s native ability to predict several tokens at once. Unsloth embeds the MTP tensor directly into the NVFP4 model, so no extra module is required.

FP8 KV‑Cache Calibration

FP8 KV‑Cache calibration doubles the context length, which is especially beneficial for agentic workflows that need to remember long histories.

Performance Benchmarks

Throughput (1× B200, 128 concurrency)

Qwen3.6‑27B NVFP4 : 2.5× faster than NVIDIA’s NVFP4.

Qwen3.6‑35B‑A3B NVFP4 (standard) : 1.56× speed‑up.

Qwen3.6‑35B‑A3B NVFP4‑Fast : 1.79× speed‑up (full W4A4).

High‑concurrency 35B‑A3B can reach 17,561 tokens/s .

Decode speed improves by 1.03× for 27B and 1.17–1.22× for 35B‑A3B.

Accuracy Comparison (near‑lossless)

Benchmarks show Unsloth’s NVFP4 matches or exceeds BF16:

Qwen3.6‑27B

Unsloth NVFP4 – MMLU‑Pro: 86.25 , GPQA: 86.34 , AIME 2025: 93.12

NVIDIA NVFP4 – MMLU‑Pro: 85.96 , GPQA: 86.87 , AIME 2025: 93.12

FP8 – MMLU‑Pro: 86.11 , GPQA: 86.87 , AIME 2025: 93.75

BF16 – MMLU‑Pro: 85.96 , GPQA: 88.13 , AIME 2025: 93.33

Qwen3.6‑35B‑A3B

Unsloth NVFP4 – MMLU‑Pro: 85.85 , GPQA: 86.74 , AIME 2025: 92.29

Unsloth Fast – MMLU‑Pro: 85.58 , GPQA: 87.75 , AIME 2025: 91.67

NVIDIA NVFP4 – MMLU‑Pro: 85.60 , GPQA: 87.12 , AIME 2025: 91.88

FP8 – MMLU‑Pro: 85.75 , GPQA: 86.74 , AIME 2025: 93.12

BF16 – MMLU‑Pro: 85.75 , GPQA: 86.36 , AIME 2025: 92.50

Version Choice for 35B‑A3B

NVFP4 (recommended) : partial high‑precision layers for better accuracy, 1.56× speed‑up.

NVFP4‑Fast : full W4A4 for maximum speed, 1.79× speed‑up.

Use the standard version for agent or tool‑calling workloads; choose the Fast version for pure chat or latency‑sensitive scenarios.

Hardware Requirements

✅ RTX 50 series (5090, 5080, …)

✅ DGX Spark

✅ B200 / B300

❌ RTX 40 series and older – not supported

If a Blackwell GPU is unavailable, Unsloth’s Dynamic GGUF quantization combined with MTP still delivers good performance on older cards.

How to Run

Option 1: vLLM (recommended)

# Install vLLM
uv venv myenv --python 3.12
uv pip install --python myenv/bin/python \
    "vllm==0.24.0" "nvidia-cutlass-dsl==4.5.2" --torch-backend=auto

# Launch 27B NVFP4
vllm serve unsloth/Qwen3.6-27B-NVFP4 \
    --speculative-config '{"method": "mtp", "num_speculative_tokens": 2}'

# Launch 35B‑A3B Fast version
vllm serve unsloth/Qwen3.6-35B-A3B-NVFP4-Fast \
    --speculative-config '{"method": "mtp", "num_speculative_tokens": 2}'

Option 2: SGLang

python -m sglang.launch_server \
    --model-path unsloth/Qwen3.6-27B-NVFP4 \
    --speculative-algorithm NEXTN \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4

Option 3: Unsloth Studio (simplest)

# Install
curl -fsSL https://unsloth.ai/install.sh | sh

# Start
unsloth studio -p 8888

Open a browser, search for Qwen3.6, download the model, and the UI will automatically select optimal inference parameters, including MTP.

Deploy as an API Service

from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:8000/v1",
    api_key="sk-no-key-required",
)

completion = client.chat.completions.create(
    model="unsloth/Qwen3.6-27B-NVFP4",
    messages=[{"role": "user", "content": "帮我写一个 Snake 游戏"}],
    temperature=0.6,
    top_p=0.95,
    extra_body={"top_k": 20},
)
print(completion.choices[0].message.content)

This endpoint is compatible with any OpenAI‑compatible client (Claude Code, Codex, etc.).

Comparison with Other Solutions

Speed : Unsloth NVFP4 ★★★★★, NVIDIA NVFP4 ★★, Unsloth GGUF + MTP ★★★, Ollama ★★.

Accuracy : Unsloth NVFP4 ★★★★★, NVIDIA NVFP4 ★★★★, Unsloth GGUF + MTP ★★★★, Ollama ★★★.

Ease of Use : Unsloth NVFP4 ★★★★, NVIDIA NVFP4 ★★★, Unsloth GGUF + MTP ★★★★★, Ollama ★★★★★.

GPU Requirement : NVFP4 variants need Blackwell; GGUF + MTP and Ollama run on any GPU.

Recommended Scenario : NVFP4 for high‑throughput production, GGUF + MTP for personal/local development, Ollama for entry‑level experience.

Final Remarks

The core innovation of Unsloth’s Dynamic NVFP4 is the W4A4 scheme that compresses both weights and activations to 4‑bit and leverages the Blackwell FP4 Tensor Core for native computation, bypassing the de‑quantization bottleneck of traditional approaches.

For users with Blackwell GPUs, this is currently the fastest way to run Qwen3.6: the 27B model fits in 24 GB VRAM and matches BF16 accuracy. Users on older GPUs should consider the Dynamic GGUF + MTP path.

Important note: avoid CUDA 13.2 (it produces garbled output); use a version below 13.2 or 13.3 instead.