Running Qwen3.8-27B on a Single RTX PRO 6000 and Hooking It Up to DeepSeek Harness

When Alibaba released the open‑source Qwen3.8‑27B model with a native context length of 262 K and Multi‑Token Prediction (MTP), the author immediately downloaded the ~52 GB model consisting of 18 safetensors shards.

The available hardware is an NVIDIA RTX PRO 6000 workstation GPU with 96 GB of VRAM, which can load the model in native BF16 precision without any quantization. The same machine previously ran Qwen3‑32B using a vLLM Docker image; the environment includes GPU 0 (RTX PRO 6000), driver 580.159.03, CUDA 13.0, vLLM 0.18.0, PyTorch 2.10.0+cu128, CUDA 12.8, and Transformers 4.57.6.

模型软链接
qwenmodel -> Qwen/Qwen3.8-27B

容器启动参数
--model /model
--served-model-name qwenmodel
--dtype bfloat16
--max-model-len 16384
--max-num-seqs 32

After linking the new weights via a symlink and launching the container with the same Dockerfile, the model loads in 22 seconds, consuming 51.08 GiB of VRAM and leaving about 28.27 GiB for KV cache. vLLM identifies the architecture as Qwen3_5ForConditionalGeneration and the API works.

A simple coding request for a thread‑safe LRU cache with max_tokens=4096 returns a finish_reason=length without any code output, illustrating the default adaptive‑thinking level xhigh in the vLLM recipe.

Performance testing starts with a baseline of 29.11 tok/s for a 1 K input / 1 K output request. Enabling MTP yields:

MTP=1: 42.33 tok/s

MTP=2: 51.14 tok/s

MTP=3: 54.54 tok/s

With 8 concurrent requests, throughput rises from 191.66 tok/s to 295.84 tok/s; with 32 concurrent requests, it rises from 561.89 tok/s to 723.13 tok/s. Acceptance rate at MTP=3 drops to ~32%, so the author keeps MTP=3 as the final setting.

To push the context length to the native 262 K limit, the author first tries FP8 KV cache to save memory, but short‑request speed falls from 54.54 tok/s to 25.85 tok/s and acceptance drops from 52.51 % to 7.05 %. Reverting to BF16/auto KV while keeping --max-model-len 262144 restores short‑request speed to 52.79 tok/s and yields an estimated full‑length concurrency of 1.56×.

--dtype bfloat16
--max-model-len 262144
--max-num-seqs 32
--gpu-memory-utilization 0.90
--speculative-config '{"method":"mtp","num_speculative_tokens":3}'

With the model stable, the author integrates it into DeepSeek Harness, which provides an OpenAI‑compatible API via vLLM and adds tool‑calling capabilities. The custom provider qwenlocal is configured with:

Provider ID    qwenlocal
Model ID       qwenmodel
Context         262144
Max Output      32768

Additional vLLM flags enable reasoning parsing, tool‑call parsing, and automatic tool choice:

--reasoning-parser qwen3
--enable-auto-tool-choice
--tool-call-parser qwen3_coder

When a new session selects the local Qwen3.8‑27B model, it correctly identifies itself as “Qwen” and acknowledges the coding‑agent environment, capable of reading, editing, and executing code. A forced file‑discovery request returns the eight files in the working directory, confirming the closed‑loop of API, tool call, and result return.

For a concrete coding task, the model reads the project’s README and scripts, proposes a safe improvement, modifies apply-welcome-copy.mjs, and runs syntax checks. It handles duplicate‑text scenarios, adds regression tests covering 48 assertions, and completes the workflow in 13 steps, taking 4 min 20 s of model inference and 0.9 s of tool calls.

Overall, the DeepSeek Harness run logs 5 rounds and 23 steps, with 16 min 18 s of model inference, 2.7 s of tool calls, an average first‑token latency of 6.8 s, and a generation speed of about 45 tok/s. The author notes that most time is spent re‑reading context and generating, not file operations, and plans to explore prefix caching and context compression in future tests.

Reference Links

Qwen3.8-27B model card: https://huggingface.co/Qwen/Qwen3.8-27B

DeepSeek Harness repository: https://github.com/deepseek-ai/deepseek-harness

vLLM speculative decoding docs: https://docs.vllm.ai/en/latest/features/speculative_decoding/