DeepSeek V4 Pro Distilled into 9B Qwen Model Tops Hugging Face

Overview

A 9B parameter model named DeepSeek-V4-Pro-Qwen3.5-9B (later refined by xtools-at ) topped the Hugging Face trending list. The model merges DeepSeek V4 Pro's reasoning capabilities into Alibaba's open-source Qwen3.5-9B through a two-stage community effort.

Two-Stage Development

Stage 1: Distillation by Jackrong

Teacher model: DeepSeek V4 Pro (1.6T total parameters) in Max Effect mode.

Data generation: ~250,000 math and STEM samples recording step-by-step problem decomposition and final answers.

Student model: Qwen3.5-9B (9B parameters).

Training: Supervised fine-tuning (SFT) on the generated samples, followed by a lightweight reinforcement learning round using GSPO . Both stages focused exclusively on math and STEM reasoning; no coding data was used.

Stage 2: Refusal Adjustment by xtools-at

Method: Heretic v2.0.0.dev0 with its ARA (Activation-based Refusal Adjustment) technique.

Mechanism: Compares internal activations across prompt types to isolate refusal-related signals, then modifies weights in layers 9–24 to weaken those signals.

Result: On 100 test prompts, the original distilled version refused all 100; the adjusted version refused only 3 . KL divergence between pre- and post-adjustment output distributions was reported as 0.0422 .

Benchmark Results

GSM8K (NVFP4 quantized, 4 runs)

Distilled 9B model: 94.5% average accuracy.

Reference: DeepSeek V4 Pro 92.6% , Llama 3.3 70B Instruct 94.84% .

MMLU-Pro (500 questions each in math, physics, chemistry)

Math: 90.60% → 92.40% (+1.80 pp).

Physics: +0.4 pp .

Chemistry: +0.6 pp .

Reasoning Token Efficiency (tokens per correct answer)

Math: 47.4% reduction .

Physics: 30.8% reduction .

Chemistry: 32.1% reduction .

The gains are uneven across subjects, with math showing the largest accuracy and efficiency improvements.

Format Compliance

Prompts required the model to output only a single uppercase letter after reasoning. Compliance rates:

Distilled 9B model: 97.2% .

Original Qwen3.5-9B: 37.3% .

Claude Mythos distilled 9B: 69.6% .

Quantization & Local Deployment

Community developer Sakura721 converted the xtools-at derivative to Q4_K_M GGUF (4-bit quantization).

Runs locally via LM Studio , Ollama , or llama.cpp .

For image input, the corresponding mmproj file from the same repository is required to encode images into model-compatible embeddings.

Limitations & Next Steps

The refusal adjustment results are reported separately from the original science benchmarks. The article notes that re-running Jackrong's exact science test set on the final derivative is needed to confirm how much of the distilled accuracy and token efficiency is retained after the ARA weight modifications. Coding and tool-use improvements are mentioned as preliminary observations with a playable Star Skip demo, but no formal code/Agent benchmarks are provided yet.