Qwen 4 Imminent: Leaked Code & Anonymous Models Reveal Architecture Details

Alibaba's next-generation Tongyi Qianwen model, Qwen 4, has not been officially announced, but a series of community discoveries indicate its release is approaching.

Inference Code Optimizations Appear

On October 23, the Intel-maintained vLLM repository merged an optimization for Qwen4-27B covering FP8 and INT4 quantization inference for single-GPU and dual-GPU configurations. The code describes the model as dense, but developers note they lack the sparse attention weights for end-to-end testing.

Shortly after, the open-source inference project llama.cpp opened an issue titled "Get ready for qwen4", signaling preparation for support.

TensorFold further began developing native CUDA kernels for Qwen4-27B targeting NVIDIA GB10 and DGX Spark platforms, adapting architecture based on leaked kernel information.

Leaked Architectural Details

From the surfaced code and community analysis, Qwen4-27B likely uses a hidden dimension of 5120, 24 query heads, 4 key-value heads, and introduces 4 residual streams. Its feed-forward network dimension may be 14336. The model may employ multi-token prediction (MTP) to generate 4 draft tokens and adopt QSA sparse attention among other new designs.

Official Hints at Dual Release

On October 10, the official Qwen X account posted an image of two capybaras with the caption "big or small?" and later commented "alright guys, let's do both!", fueling speculation that both a 27B model and a larger flagship Max variant will launch together.

Anonymous Arena Models Kalmia and Kerria

Two anonymous models, Kalmia and Kerria, appeared on the LMSYS Chatbot Arena. Kalmia emerged on October 21 and showed strong performance in code generation and 3D interactive applications. User Unicorn shared a pirate-ship Three.js demo generated by Kalmia that ran without DOM leaks or Canvas crashes, reportedly outperforming Kimi 3.1 Pro. Another developer, after 10 tests, found Kalmia's 3D generation promising but noted DeepSeek V4.1 Flash still performed better in some cases, estimating Kalmia roughly on par with MiniMax 3.1 Flash.

Kerria appeared shortly after Kalmia. Tests show it exhibits similar behavior on special tokens characteristic of the Qwen family, leading to speculation it is a newer checkpoint with improved performance. User Synopsis observed Kerria appears to be a post-Kalmia checkpoint with performance gains. User Jaycee demonstrated creative writing from Kerria, comparing it favorably against Fable 5, and another user praised a Hogwarts-themed Three.js page generated by Kerria, noting its superior music taste and soundtrack quality.

As code clues and anonymous model evaluations accumulate, anticipation for Qwen 4 grows. Whether large and small models release simultaneously and their exact capabilities remain to be officially confirmed.