DeepSeek V4, released in April 2026, introduces two versions—Pro and Flash—with up to 1.6 trillion parameters and a million‑token context window, leveraging hybrid attention, compressed KV cache, and specialized training techniques to dramatically cut hardware dependence and inference cost.
DeepSeek-V4 introduces a hybrid attention architecture, manifold‑constrained hyper‑connections, and the Muon optimizer to cut inference FLOPs and KV cache dramatically, enabling open‑source models to handle million‑token contexts at a fraction of the cost of leading closed‑source services while matching their performance.
The DeepSeek‑V4 preview details two model variants—Pro and Flash—with trillion‑scale parameters, outlines benchmark scores that surpass or match leading overseas models across code generation, real‑world fixes, engineering tasks, and world knowledge, and explains core innovations, pricing, API endpoints, and open‑source licensing.
DeepSeek-V4 introduces two open‑source LLMs—V4‑Pro with 1.6 trillion total parameters and V4‑Flash with 284 billion—offering a 1 million‑token context window, hybrid attention, multi‑head compression, and a new Muon optimizer, all under an MIT license that rivals top closed‑source models.
Alibaba’s open‑source Qwen3.6‑27B model can be run on consumer hardware with as little as 18 GB of RAM using 4‑bit quantization, and its hybrid attention architecture delivers higher accuracy on coding benchmarks such as Terminal‑Bench 2.0 and SWE‑bench Pro than the much larger 397‑B‑parameter Qwen3.5‑397B‑A17B MoE model.
A joint Moonshot AI and Tsinghua study shows that the Prefill-as-a-Service (PrfaaS) architecture, enabled by hybrid‑attention models that shrink KVCache size, can offload long Prefill work to a remote cluster and, with dual‑timescale scheduling, achieve a 54% throughput gain over homogeneous PD deployment and 32% over naive heterogeneous setups.
Google’s newly released Gemma 4 series delivers a range of open‑source LLMs—from 2.3 B to 31 B parameters—optimized for edge devices through per‑layer embeddings, mixed‑expert MoE, hybrid attention, and extensive hardware support, achieving top‑tier benchmark scores while running efficiently on phones and IoT.
In just two days, Baidu Baige and Kunlun's engineers extended the vLLM‑Kunlun Plugin to overcome asymmetric KV dimensions and integrate SWA+Sink attention, achieving lossless, high‑performance inference of the MiMo Flash V2 model on the Kunlun P800 XPU.
A recent Hugging Face pull request reveals Alibaba’s upcoming Qwen3‑Next series, highlighting its extreme‑context, parameter‑efficient design that combines a 1:50 high‑sparsity MoE, a hybrid attention architecture mixing gated attention with Gated DeltaNet, and a Multi‑Token Prediction technique, promising ten‑fold throughput gains for 32K‑plus token contexts.
MiniMax’s latest M1 model, unveiled after a $300 million funding round, showcases a 4.56‑trillion‑parameter hybrid‑expert architecture with lightning attention, supporting up to one million tokens, and leverages reinforcement‑learning techniques to enhance long‑context handling, inference efficiency, and system‑2 reasoning capabilities.
