How Qwen3.5 Packs 397B Parameters Yet Activates Only 17B – A Deep Dive into Its Multimodal Architecture

Unified Sensory Fusion

Traditional multimodal models stack a visual encoder on top of a language model, converting images to feature vectors before feeding them to the language component. Qwen3.5 introduces early text‑vision fusion during pre‑training, learning a shared representation space that eliminates the need for a separate visual encoder pipeline. This end‑to‑end design enables tasks such as spatial reasoning, GUI manipulation, and video understanding without intermediate information loss.

Hybrid Architecture for Scale and Efficiency

Qwen3.5 builds on the Qwen3‑Next design by combining linear attention (Gated Delta Networks) with a sparse mixture‑of‑experts (MoE) layer. The model contains 397 B total parameters but activates only 17 B parameters per forward pass, preserving knowledge while dramatically reducing compute cost.

Additional optimisations include:

Gated DeltaNet + Gated Attention for faster long‑context processing.

Multi‑token prediction, allowing the model to generate several tokens in a single forward step.

Decoding throughput is 8.6× higher than Qwen3‑Max at a 32 k context length and 19× higher at 256 k, demonstrating strong scalability.

Cross‑Language Coverage and Asynchronous RL

Data filtering has been tightened, resulting in higher‑quality Chinese, English, and STEM corpora. Language coverage expands from 119 to 201 languages, including many low‑resource dialects. The token vocabulary grows to 250 k, improving encoding/decoding efficiency for long‑tail languages by 10‑60 %.

An extensible asynchronous reinforcement‑learning (RL) framework continuously refines the model. The framework supports the full‑size model across text, multimodal, and multi‑turn interaction scenarios and uses a decoupled training‑inference design to maximise hardware utilisation.

Heterogeneous Compute and Interaction Redesign

Native multimodal training on heterogeneous hardware decouples visual and language components, enabling parallel execution and sparse activation that overlaps module computation. FP8 pipelines process activations, routing, and matrix multiplications in low precision, while sensitive layers retain BF16 for numerical stability.

Key outcomes:

~50 % reduction in activation memory.

>10 % increase in inference speed.

Scalability to tens of trillions of tokens.

Benchmark Performance and Availability

On benchmarks such as BFCL‑V4, VITA‑Bench, DeepPlanning, Tool‑Decathlon, and MCP‑Mark, Qwen3.5 ranks among the top open‑source models, surpassing trillion‑parameter competitors (e.g., GPT‑5.2, Claude 4.5 Opus) on visual‑language tasks, mathematics, and reasoning.

Model weights and code are publicly available:

Blog post: https://qwen.ai/blog?id=qwen3.5

GitHub repository: https://github.com/QwenLM/Qwen3.5

Hugging Face hub: https://huggingface.co/Qwen/Qwen3.5-397B-A17B

ModelScope page: https://modelscope.cn/models/Qwen/Qwen3.5-397B-A17B

Conclusion

Qwen3.5 demonstrates that a hybrid architecture combined with native multimodal training and an asynchronous RL framework can reconcile massive parameter scales with practical compute efficiency, setting a new benchmark for future AI research.