Testing Qwen3.8-27B on Dual 4090 GPUs and Connecting to Claude Code for Unlimited Tokens

Model overview

Qwen3.8‑27B is a 27.8 B dense multimodal model released by Alibaba. It supports text, image and video inputs, provides a native 256 K context window, and is licensed under Apache 2.0. The model targets coding and agent capabilities; the official SWE‑bench Pro score is 61.7.

Hardware and model download

The deployment used a server with two turbo‑variant RTX 4090 GPUs, each with 48 GB VRAM (96 GB total). BF16 weight files occupy ~56 GB; tensor‑parallel size 2 splits the model to ~28 GB per GPU, leaving the remainder for KV cache. The model was downloaded from Hugging Face / ModelScope as 18 safetensors shards, completing in roughly half an hour on the internal network.

Software stack constraints

The GPU driver (560) caps CUDA at 12.6, while vLLM 0.20+ requires CUDA 13. Therefore vLLM was pinned to the 0.19.x series to match the driver.

vLLM service launch

vllm serve models/Qwen3.8-27B \
  --served-model-name qwen3.8-27b \
  --tensor-parallel-size 2 \
  --max-model-len 262144 \
  --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.92 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --api-key <填写api-Key> --host 0.0.0.0 --port 8001

Performance observations

Weight loading took about seven minutes; each GPU used 45.2 GB after loading. Single‑stream decoding achieved 20‑30 tokens / s, sufficient for casual conversation but too slow for production workloads.

Configuration pitfalls

The default tool-call-parser (hermes JSON) produced plain‑text tool calls. Switching to the model‑specific qwen3_xml parser yielded proper tool_use blocks required by Claude Code.

Setting --max-model-len to 128 K was insufficient for many tasks. With BF16 precision the KV cache held only ~133 k tokens, so a 256 K request could not fit. Changing the KV cache datatype to FP8 doubled capacity to ~268 k tokens, enabling the full 256 K window at the cost of reduced precision.

Integration with Claude Code via ccmr

The model was routed to Claude Code using the ccmr gateway, which implements the Anthropic /v1/messages endpoint directly. Repository: https://github.com/luwill/Claude-Code-Model-Router

qwen-local:
  base_url: http://<内网IP>:8001
  api_key_env: QWEN_LOCAL_API_KEY
  auth_header: Authorization   # vLLM only accepts Bearer
  auth_type: bearer
  variants:
    27b:
      model_id: qwen3.8-27b
      max_tokens: 32768
      context_window: 262144

Running the model through Claude Code:

ccmr claude --model qwen-local-27b

Test case

A test where Claude Code + Qwen3.8‑27B generated a front‑end design for a personal health‑management app produced reasonable output but was painfully slow, leading to an early abort. Consumer‑grade GPUs therefore cannot handle production‑level tasks.

Benchmark scores

Community benchmarks give the model a score of 52 on the Artificial Analysis suite, comparable to much larger models such as GLM 5.2 and GPT 5.6 Luna, and matching DeepSeek Flash 0731 (284 B) within the Qwen series. The official SWE‑bench Pro score is 61.7; Claude Opus 4.6 Max scores 53.4 on a different harness, serving only as a reference point.

Community variants

Various quantized and uncensored variants are reported to run on 16 GB Macs.

Token generation characteristics

Compared with its predecessor Qwen3.6, Qwen3.8‑27B generates roughly twice as many tokens for the same task, resulting in slower generation (20‑30 tok/s) and longer “thinking” phases. World‑knowledge performance is slightly lower, but non‑hallucination scores are higher, indicating better self‑awareness of uncertainty and reliance on tool calls.

Conclusion

Local deployment of large models is valuable for scenarios where data cannot leave the internal network, the Apache 2.0 license permits commercial use, and unlimited token context is desired. It is feasible on high‑memory professional GPUs, but most users are better served by cloud‑hosted offerings.