NVIDIA's 4B Draft Model Achieves 6.13 Tokens/Step for Qwen3-27B: Hardware Reality Check
NVIDIA released three draft models for speculative decoding with Qwen3.8-27B, with the 4.21B model achieving 6.13 accepted tokens per step—59% higher than native MTP—but acceptance length isn't end-to-end throughput, and local deployment requires careful memory planning, especially for Mac and single-GPU users.
What NVIDIA Released: Three Draft Models for Qwen3.8-27B Speculative Decoding
On November 17, NVIDIA published three external draft models on Hugging Face designed for speculative decoding with Qwen3.8-27B. Speculative decoding works in three steps: (1) a small draft model quickly generates candidate tokens, (2) the large target model verifies them in parallel, (3) rejected positions are resampled. Because rejection sampling preserves the target model's output distribution, the process is "lossless" at the sampling-distribution level—not a smarter model.
The Three Draft Models
Qwen3.5-4B Draft (4.21B params): shares Qwen3.8-27B's tokenizer and 248,320 vocabulary; highest acceptance length.
Qwen3.5-0.8B Draft (0.75B params): ~1/5 the size of 4B, yet achieves 4.09 acceptance length.
Llama-3.2-1B Draft (1.48B params): uses a rebuilt aligned vocabulary; acceptance length 3.52, lowest of the three.
Training used 199,998 single-turn dialogues (~1.079B tokens), recording the target model's top-20 output distribution at each reply position for alignment. The 4B draft trained for 1 epoch; the other two for 2 epochs.
Official Acceptance-Length Numbers (Not End-to-End tok/s)
Experiments ran on the SPEED-Bench qualitative subset (880 first-turn dialogues) at temperature=1, top-k=20, top-p=0.95.
Fixed draft block of 16: 4B draft = 5.58 accepted tokens/step; native MTP = 3.81.
Unbounded draft length: 4B draft = 6.13; native MTP = 3.85.
4B exceeds MTP by 59%; 0.8B draft reaches 4.09 (still above MTP's 3.85).
Llama-3.2-1B scores 3.52, below MTP under fixed length.
Critical caveat: The article's 100K-context speed chart provides no concrete tok/s figures. Qwen3.8-27B results are from NVIDIA internal measurement; Gemma 4 31B results from Artificial Analysis. No independent third-party end-to-end throughput numbers exist yet. Higher acceptance length indicates greater acceleration potential, not guaranteed proportional speedup.
Three Metrics Not to Conflate
Acceptance length, end-to-end tok/s, and local runnability are distinct. The model cards state checkpoints are standard Hugging Face models loadable in vLLM, SGLang, and TRT-LLM. However, NVIDIA's recommended deployment target is Groq 3 LPU/LPX runtime, which features pipeline verification, parallel drafting, and dynamic unbounded drafting—mechanisms that differ from typical GPU-based vLLM setups.
Practical Guidance by User Segment
Mac / llama.cpp users: No official validation; "loadable" ≠ "accelerated."
Single 24GB/32GB GPU users: First verify the 27B base model fits; the draft adds extra weight and KV cache overhead.
Existing vLLM/SGLang users: Wait for community to stabilize external-draft parameters, then run A/B tests on identical workloads—don't rely on screenshots.
Groq 3 LPX environments: This is the native, optimal deployment target.
Next Steps: Check Your Hardware First
The author's ModelFit Chinese site (linked at article top and via "Read Original") lets you select device and memory to see if qwen3.8-27b-q4 fits and estimate speed ranges. Core advice: clear the "fits in memory" hurdle before chasing acceleration . If 27B Q4 is unstable on your machine, the 4B draft only adds memory pressure. Conversely, on 64GB/128GB systems already running 27B Q4, it's worth monitoring upcoming real-world benchmarks.
Final question to readers: What single-stream tok/s are you measuring for 27B today? What hardware and runtime? Share configs and numbers in comments for comparison with ModelFit estimates.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
