Tongyi's Qwen4-Max reduces per-task cost from $4.43 to $1.67 (62% less) on its Work360 benchmark while raising scores from 47.74 to 52.69, using a 2.4T-parameter architecture with 710M activated parameters, phrase embeddings, gated recurrent layers, mixture-of-experts, and sparse attention, with training loss declining steadily without spikes.
Tencent's Hy4 Flash, a 224B MoE model with 12B active parameters, achieves 150+ tokens/s decoding speed at 1/20 the cost of its predecessor, outperforms DeepSeek-V4.1-Flash on cost efficiency, and matches Claude Haiku 5.5 user satisfaction in blind tests across 259 office tasks.
This roundup covers seven high-quality trending GitHub projects: a categorized public API directory, an AI agent chat manager integrating Claude and Codex, a game studio template with 49 agent roles, offline voice dictation tools, a lightweight headless browser for AI automation, a local MoE model inference engine, and a dictation app with custom vocabulary corrections.
The article details the first public case of applying recursive self-improvement (RSI) closed loops to full frontier-scale pre-training, producing the 801-30B-A3B MoE model trained on 11.8T tokens, which outperforms Qwen3-30B-A3B and Nemotron-3.5-Lightning-30B-A3B through three types of feedback loops and agent-generated training data.
In a Stanford HAI dialogue, Jeff Dean and Fei-Fei Li distinguish self-improvement from recursive self-improvement (RSI), argue RSI has physical limits due to real-world compute constraints, outline Discovery Loop's focus on ML research, engineering, and hardware design, discuss AI's role in autonomous scientific discovery and mathematical verification, and reflect on Transformer, distillation, MoE, and code capabilities as key AI milestones.
Shanghai AI Lab and Xinxu AI introduce No Second Run, a recursive self-improvement method for LLM pretraining using debugging, headroom evaluation, and experience-to-metric loops. In three months they trained the 801-30B-A3B MoE model (30B total, 3B active params) on 11.8T tokens, scoring 73.13 average on 13 base benchmarks (second only to Qwen3.5-35B-A3B) and achieving top Agent benchmark results after 6.33B-token SFT without RL.
ETH Zurich postdoc Bingcong Li will join HKU as a tenure-track assistant professor, focusing on efficient scaling of large language models through architecture design and optimization, including width/depth scaling, LoRA fine-tuning, MoE efficiency, and zero-order optimization methods like DPZero and RefLoRA.
Runway released Praxis-1, a 27B-parameter mixture-of-experts diffusion model that jointly predicts video frames and robot actions, achieving 46.2% success on the RoboLab-120 benchmark over 1,200 trials, outperforming FLUX 3 Action, Cosmos3 Nano, and π0.5, with weights, inference, and fine-tuning code open-sourced under Apache 2.0 and deployable on hardware from H100 to Jetson Thor.
The article explains how llama-modchip runs 284B DeepSeek-V4-Flash on a 64GB RAM CPU-only machine via aggressive quantization (1.71 bits/weight), while a 96GB Xeon server fails due to tensor format mismatch and missing AVX-512. It breaks down inference latency for long contracts (~10 minutes for 20k tokens) and proposes three evaluation metrics: time to first token, key-field extraction accuracy, and total processing time.
Researchers from Zhejiang University, ByteDance Seed, and Tsinghua University propose Cortex, a method that consolidates contextual experience into model parameters via prospective rehearsal and context distillation, enabling agents to retain skills and knowledge without external retrieval, validated across tool use, fact updating, and agent tasks with minimal parameter growth.
Datawhale community launches Diy-LLM, a Chinese-language tutorial built on Stanford CS336 covering tokenizers, Transformer architecture, MoE, distributed training, scaling laws, inference, alignment, and multimodal models through 16 chapters and 6 practical assignments with code.
IQuest Research releases technical report for IQuest-Q1, a 320B MoE model with 15B active parameters, 512K context window, recursive multi-token prediction, and agent-focused post-training, achieving strong benchmarks across coding, tool use, and reasoning tasks.
Recursive_SI's first model Flint-40B-A2B, a 40B MoE model activating 2.5B parameters per token, was built in under four months using the automated Eureka Machine system. It achieves state-of-the-art local coding performance on benchmarks like SWE-Bench while running at 125+ tokens/sec on MacBook Pro M4 Pro and 150+ tokens/sec on NVIDIA RTX Spark, with open weights on Hugging Face under Apache 2.0.
Recursive team open-sources Flint-40B-A2B, a 40B-parameter MoE model optimized for local coding agents, developed in under four months using their Eureka Machine AI research system; it achieves over 125 tokens/sec on MacBook Pro and demonstrates autonomous research capabilities on tasks like circle packing, factory scheduling (26% improvement), and bike-sharing demand prediction (30.8% RMSLE reduction).
French AI company Mistral releases Mistral Large 4, a 1.05 trillion parameter open-source MoE model with 128 experts and 180B active parameters, achieving 91.2 MMLU-Pro and 89.7% HumanEval+, detailing architecture, training on 16K H200 GPUs, inference via expert offloading, and Apache 2.0 licensing enabling sovereign AI deployment.
Ant Group's Bailin team open-sources Ling-3.1-flash, a 560B-parameter mixture-of-experts model that activates 25B parameters per token, supports 1M context, and achieves 210 tokens/sec output speed with an intelligence index of 41 on Artificial Analysis, while quantization reduces weight from 534GB to 292GB enabling deployment on 4 Hopper GPUs instead of 8.
This article chronicles DeepSeek's paper timeline from 2023 to 2026, analyzing how each model generation optimized training compute, KV cache, and throughput via MoE, MLA, sparse attention, and conditional memory, shifting long-context competition from maximum length to per-token byte cost.
Qwen researchers propose ID Balancing, a PID-based load control method for extremely sparse MoE models, achieving stable expert load with 768 experts and Top-3 routing, reducing MaxVio by over 50% and MinVio by 12% while maintaining competitive language modeling and downstream task performance.
An independent researcher reverse-engineered TypeSafe's closed-source Jev classification model using 10,000 API calls, revealing its architecture uses a classification head with shared prefix inference, listwise scoring, and exceptional calibration (ECE 0.0313), challenging assumptions about black-box security.
At the 2026 Yunqi Conference, Banma Smart unveiled AutoOmni 2.0-23B-A3B, a sparse MoE edge model that achieves near-cloud performance on vehicle-grade chips, addressing memory, latency, and concurrency constraints while enabling privacy-preserving, low-cost AI with certified safety frameworks for mass production.
The article benchmarks StepFun's Step 5 Preview against DeepSeek V4 Pro across three coding challenges—2D animation, token bucket demo, and CSV analysis workbench—revealing Step 5 Preview delivers richer details and better test coverage despite longer generation times, with per-task cost at just 35% of GLM-5.3 and 12.5% of Claude Opus 5.
This article details how a custom C inference engine runs the 744B-parameter GLM 5.2 MoE model on 25GB RAM by quantizing routed experts to int4, streaming them from disk via pread, compressing KV cache 57x with Multi-head Latent Attention, and employing a five-tier memory hierarchy with speculative decoding.
DeepSeek V4.1 Flash introduces a new asymmetric Causal Encoder-Decoder architecture with 552B MoE parameters, reducing KV cache HBM demand by 75% and SSD demand by 87.5% while adding native multimodal vision and speculative decoding.
The SMELT framework from Tsinghua and ByteDance Seed fairly compares Looped Transformers against baselines by matching compute, parameters, and KV cache, finding that looping the middle 50% of layers twice with a larger depth-width ratio consistently reduces validation loss across scales, saves 6.8–18% training compute, and yields downstream gains beyond loss reduction.
This article details Huawei's xDeepServe system for deploying massive Mixture-of-Experts models on the CloudMatrix384 supernode, covering the XCCL communication library, FlowServe decentralized serving engine, Transformerless execution architecture with prefill-decode and MoE-Attention decoupling, and hierarchical fault tolerance, achieving 2400 tokens/s per chip at 50ms TPOT.
Xiaomi releases Xiaomi-TabLDM, a foundation model for tabular data that uses large-scale synthetic pretraining, efficient model scaling with dual-stream feature groups and sparse MoE, and test-time scaling to achieve top-tier performance on four public benchmarks and real-world industrial tasks without per-dataset retraining.
This article explains the distinction between total and activated parameters in Mixture-of-Experts (MoE) models, contrasting dense and sparse architectures, detailing expert routing mechanisms, and analyzing memory and compute implications across model loading, prefill, and decode stages.
Tencent releases Hunyuan Hy4 preview, a 770B parameter mixture-of-experts model with 49B activated parameters and 1M context length, achieving top-tier open-source performance across coding, office, gaming, and scientific tasks, with internal blind tests showing 2.99/4 score surpassing GLM 5.3 and Kimi K3, plus 31.8% inference throughput gains via self-optimization.
Ornith-1.5, an MIT‑licensed large‑model framework from DeepReinforce, introduces a self‑improving loop that autonomously generates tasks, builds scaffolds, and rolls out solutions, achieving state‑of‑the‑art performance at 9B and 35B scales and delivering benchmark scores comparable to Claude Opus 4.8 for its 397B MoE variant.
The article explains why token‑to‑expert routing in large Mixture‑of‑Experts models can become highly imbalanced, outlines the resulting compute bottlenecks, token overflow and expert degradation, and then surveys eight concrete load‑balancing strategies—ranging from importance‑based auxiliary losses to capacity‑aware token dropping and loss‑free bias control—detailing their mechanisms, trade‑offs, and practical considerations.
In early 2026, open-weight LLMs such as Llama 4, Qwen 3 and DeepSeek‑V3/R2 began matching or surpassing leading closed models like GPT‑4.5 and Claude Opus 5, driven by MoE architectures, FP8 precision, GRPO training and aggressive inference optimizations, prompting a reassessment of enterprise AI strategy.
The article breaks down Kimi K3’s 2.8‑trillion‑parameter architecture, explains its 4‑bit MXFP4 quantization, calculates the ~1.4 TB memory requirement, shows that 8‑GPU clusters (e.g., NVIDIA B300 or AMD MI350X) are needed for self‑hosting, and compares these costs with the per‑token API pricing, highlighting when each option is economical.
The paper systematically derives scaling laws for mixture-of-experts diffusion language models, trains a 30B-parameter LLaDA MoE v2 model guided by these laws, and demonstrates that it matches or exceeds state-of-the-art autoregressive models while using far less compute.
LLaDA MoE v2benchmark performancediffusion language model
ByteDance is reportedly pre‑training a 10‑trillion‑parameter AI model that dwarfs domestic rivals, uses a Mixture‑of‑Experts architecture with low activation ratios, demands roughly 36 000 Blackwell GPUs and $2.5 billion in hardware, and deliberately avoids distilling competitor models, raising questions about China’s AI frontier timeline.
In a Stanford interview hosted by Dawn Song, Jeff Dean reflects on the origins of MoE, the transformative impact of deep learning, lessons from TensorFlow, how to spot breakthrough directions, AI agent risks, and his new venture Discovery Loop that aims to automate scientific research.
The article examines K3’s architecture—combining KDA, MLA, Stable Latent MoE and AttnRes—detailing the replacement of SwiGLU with SiTU‑GLU, the addition of RMSNorm for training stability, the Quantile Balancing load‑balancing scheme, and the trade‑offs behind its MLA and NoPE attention choices.
JetBrains released the open‑source 12‑billion‑parameter Mellum2 model, a MoE‑based code AI that delivers private on‑prem deployment, high‑throughput inference, and strong code‑generation benchmarks, positioning it as a fast, specialized alternative to Claude Code and other proprietary models.
AirLLM introduces layer‑wise inference and per‑expert streaming to decouple VRAM usage from model size, enabling the 2.8‑trillion‑parameter Kimi K3 LLM to run on a single consumer‑grade GPU while preserving full‑precision accuracy and offering security‑focused insights.
Kimi K3, a 2.8‑trillion‑parameter MoE model with a 100 k‑token context, has been fully open‑sourced along with its weights, technical report and infra (MoonEP, FlashKDA, AgentEnv), delivering programming and agent benchmark results that rival top closed models such as Claude Fable 5 and GPT‑5.6 while sparking debate over alleged distillation and emphasizing AI safety and open‑weight governance.
The blog details how xHC expands language‑model residual streams to 16, achieving nearly double the gain of DeepSeek mHC on 18B and 28B MoE models, and explains the design of Temporal Feature Augmentation and Sparse Write that make large‑N scaling both effective and affordable.
Hyper-Connectionslarge language modelsmixture of experts
The article examines how Chinese LLMs such as Kimi K3 (2.8 T) and Qwen 3.8‑Max (2.4 T) achieved 2‑trillion‑parameter scales by progressively decoupling parameters from compute with Mixture‑of‑Experts, optimizing attention, and introducing training tricks like Muon and low‑precision quantization, tracing four years of architectural advances.
Model Scalingattention mechanismslarge language models
Latent MoE, demonstrated by NVIDIA’s Nemotron 3 Super and Moonshot AI’s 2.8 T‑parameter Kimi K3, compresses expert computations into a lower‑dimensional latent space, cutting memory reads and All‑to‑All traffic by fourfold, enabling more experts per token, higher accuracy, and up to 3.5× faster inference.
On July 16, the Moon‑of‑Darkness team released Kimi K3, a 2.8‑trillion‑parameter open‑source large language model that introduces mixed‑linear attention, attention residuals, and a highly efficient Mixture‑of‑Experts design, achieving roughly 2.5× the scaling efficiency of its predecessor while approaching the performance of top closed‑source models.
The article reviews Kimi K3’s 2.8‑trillion‑parameter architecture, its performance advantages over previous models, pricing details, and demonstrates how the Claude Code Model Router (ccmr) enables seamless use of K3 within Claude Code through simple command‑line steps.
Kimi K3, a 2.8‑trillion‑parameter open‑source LLM, outperforms top closed‑source models in benchmarks, excels at long‑range coding, GPU kernel optimization, and multimodal tasks, while introducing novel attention mechanisms, a compact Triton‑like compiler, and even a prototype ASIC chip.
Thinking Machines Lab unveiled Inkling, a 975‑billion‑parameter open‑weight multimodal model featuring a hybrid‑expert Transformer, 1‑million‑token context, and extensive benchmark results, alongside the lighter Inkling‑Small, with detailed architecture, training methodology, reinforcement‑learning enhancements, and practical examples of web‑app generation and tool‑calling.
In the second half of 2026, enterprise AI shifts from adoption to reliable, cost‑effective deployment, with six key trends—including multi‑agent orchestration, GraphRAG retrieval, MoE model clusters, AI observability, built‑in data governance, and reorganized AI engineering roles—guiding tech leaders toward trustworthy AI systems.
DeepSeek V4 introduces a hybrid CSA/HCA attention mechanism and a Mixture‑of‑Experts design that reduces inference cost to 27% of its predecessor, enables a 1‑million‑token context as a standard feature, and runs efficiently on both NVIDIA GPUs and Huawei Ascend NPU, dramatically lowering the barrier for Java developers to integrate large‑scale AI.
Independent developer David Ondrej released SuperGemma 26B, an uncensored fork of Google’s Gemma 4 26B that removes all safety guardrails, runs locally on consumer‑grade GPUs, and has sparked intense debate over its technical merits, deployment simplicity, and the security risks of a truly unrestricted AI model.
Colibri demonstrates that the 744‑billion‑parameter GLM‑5.2 MoE model can be executed on a pure‑CPU system using roughly 25 GB of memory by streaming expert weights from disk, int4‑quantizing dense parameters, and employing a compact C inference engine.
The paper introduces CaRE, a scalable continual‑learning framework that leverages a bi‑level routing mixture‑of‑experts to successfully train Vision Transformers on over 300 non‑overlapping tasks, outperforming existing baselines and accompanied by a new 1,000‑class benchmark, OmniBenchmark‑1K.
LingBot‑VA 2.0 introduces the industry’s first embodied‑native pre‑trained robot brain, combining causal action modeling, a sparse MoE backbone, a semantic VAE tokenizer and asynchronous foresight reasoning to achieve six‑fold inference speedup, single‑GPU deployment and a 93.6% success rate on the RoboTwin 2.0 benchmark.
LingBot-Video is an open‑source video‑generation foundation model built for embodied AI, combining a sparse‑Mixture‑of‑Experts architecture, a multi‑stage data curriculum and six‑dimensional reward learning to achieve physically consistent video synthesis that outperforms existing open‑source baselines in both visual quality and robotic relevance.
LingBot‑VLA 2.0 introduces a token‑level loss‑free MoE, dual‑query distillation, and a 60k‑hour heterogeneous dataset to achieve cross‑embodiment visual‑language‑action capabilities across 20 robot morphologies, delivering superior benchmark performance and sub‑130 ms inference while being fully open‑sourced.
Qwen3.6‑35B‑A3B Uncensored’s jailbreak edition uses a Mixture‑of‑Experts architecture to activate only ~3 B parameters, allowing the 35 B model to run on consumer GPUs with as little as 6 GB VRAM, and supports multimodal vision via llama.cpp with detailed deployment steps.
The paper identifies routing drift and expert drift as the two main causes of forgetting in multimodal continual instruction tuning (MCIT) and proposes SAME, which combines spectral‑aware routing, curvature‑aware scaling, and adaptive expert activation to keep MoE models stable, efficient, and less forgetful across long task sequences.
DiffusionGemma, an experimental discrete text diffusion model built on the 26B MoE Gemma‑4 architecture, generates whole 256‑token blocks with bidirectional attention, moving the inference bottleneck from memory bandwidth to GPU compute, achieving up to four‑fold speed gains on H100 and RTX 5090 GPUs, though with lower output quality than standard autoregressive models.
In early 2026, open‑source local large language models become practical alternatives thanks to sparse attention, MoE routing, latent KV compression, multi‑token prediction, and 4‑bit quantization, while hardware memory shortages and benchmark gaps with closed‑source models shape their deployment choices.
The newly released Kimi K2.7 Code, a 1‑trillion‑parameter mixture‑of‑experts model that activates only 32 B parameters per inference, offers a 256 K context window, supports multimodal input, improves benchmark scores by up to 31.5 % over K2.6, reduces inference token usage by about 30 %, and achieves an 81.1 MCP tool‑call score surpassing Claude Opus 4.8, while providing a CLI installation command and usage guidelines.
Google’s open‑source DiffusionGemma model leverages a 26‑billion‑parameter Mixture‑of‑Experts architecture and discrete diffusion decoding to generate whole text blocks, achieving up to four times faster generation—over 1100 tokens/s on an NVIDIA H100 and 700 tokens/s on an RTX 5090—while activating only 3.8 billion parameters during inference.
DiffusionGemma, Google's new 26‑billion‑parameter Mixture‑of‑Experts model, replaces token‑by‑token autoregression with a diffusion‑style output head that generates whole text blocks, delivering up to four‑fold speed gains on consumer GPUs while offering bidirectional attention and self‑correction, albeit with lower quality than standard Gemma 4.
The article details how the open‑source Flash‑MoE engine enables the 400B‑parameter Qwen3.5‑397B‑A17B mixture‑of‑experts model to run on an iPhone 17 Pro, achieving about 0.6 tokens per second through a custom Metal pipeline, GCD‑driven SSD streaming, and aggressive caching strategies.
The article presents Token Superposition Training (TST), which temporarily averages embeddings of non‑overlapping token bags and predicts groups of tokens in a first phase before reverting to standard token‑wise prediction, achieving up to 2.5× pre‑training speedup on 10B‑1B MoE models without altering model architecture or inference.
DeepSeek V4 introduces a suite of architectural breakthroughs—including mixed‑expert MoE, manifold‑constrained hyper‑connections, CSA/HCA hybrid attention, and FP4 quantization—that slash inference cost by up to tenfold while delivering million‑token context, competitive benchmarks, dual model variants, and a disruptive pricing strategy.
Cohere’s Command A+ model, built by Transformer co‑author Aidan Gomez and backed by Nick Frosst, packs 218 billion parameters but activates only 25 billion at inference, uses a lossless 4‑bit quantization scheme, offers native citation support, runs on a single B200 or two H100 GPUs, and is released under an Apache 2.0 license, marking a major shift toward truly open‑source, enterprise‑ready large language models.
The paper presents MP‑MoE, a new Mixture‑of‑Experts architecture that replaces top‑k routing with Mahalanobis‑based ensemble pruning, explicitly encouraging expert diversity via a co‑occurrence matrix, and uses an efficient greedy algorithm with incremental Cholesky updates, achieving higher performance with minimal training overhead and no inference cost.
SORT re‑architects industrial‑scale ranking by shifting to a request‑centric data paradigm, integrating sparse and MoE optimizations into a Transformer backbone, and delivering significant CTR‑AUC, FLOPs, and online metric improvements while maintaining high training and inference efficiency.
Tabular anomaly detection traditionally requires training a separate model for each dataset (one‑for‑one), but the new OFA‑TAD framework trains once on multiple source tables and directly transfers to unseen target tables without fine‑tuning, leveraging multi‑view distance encoding, MoE fusion, and synthetic pseudo‑anomalies to achieve state‑of‑the‑art performance across 34 datasets in 14 domains.
The article dissects DeepSeek's MoE model‑parallel strategy, explaining how GPU compute and communication are overlapped through expert, pipeline, and ZeRO‑1 parallelism, and introduces DualPipe and Waved‑EP kernels that enable efficient training on large‑scale hardware.
DeepSeekGPU Communication OverlapPipeline Parallelism
The paper introduces OFA-MAS, a one‑for‑all multi‑agent system that learns a universal topology designer using task‑aware graph encoding and a Mixture‑of‑Experts generator, achieving superior performance, OOD generalization, robustness, and efficiency across six major benchmarks.
This article breaks down the key algorithms that power large‑language models—Transformer, Mixture‑of‑Experts, Flash Attention, KV‑Cache, Multi‑Token Prediction, quantization, Chain‑of‑Thought and Retrieval‑Augmented Generation—explaining how each contributes to the performance of ChatGPT, GPT‑4 and DeepSeek.
The article introduces the open‑source Qwen3.6‑35B‑A3B model, explains its MoE architecture, three‑stage LoRA fine‑tuning, shows benchmark results where it achieves 161.9 tok/s on an RTX 5090—2.6× faster than a dense 27B counterpart—and discusses deployment tips, quantized GGUF release, and known compatibility pitfalls.
Unsloth and NVIDIA identified three low‑level bottlenecks in LLM fine‑tuning on consumer GPUs—repeated packed‑sequence metadata construction, serialized copy‑and‑compute during gradient checkpointing, and per‑expert routing overhead in MoE—and applied targeted patches that together deliver roughly a 25% speedup without changing hardware, code, or frameworks.
GPU optimizationGradient CheckpointingLLM training
The article examines Cursor's journey of shipping its Composer coding agent, detailing the agentic AI model, system architecture, and the three major production challenges—diff handling, latency accumulation, and sandbox scaling—along with the engineering solutions that enabled reliable, fast, and adoptable AI‑driven code generation.
SonicMoE, a new ultra‑fast Mixture‑of‑Experts model from Tri Dao and Ion Stoica’s team, achieves peak throughput on Nvidia Blackwell GPUs, outperforms DeepSeek’s DeepGEMM, and introduces algorithmic redesigns that decouple activation memory from expert granularity while fusing I/O‑aware kernels for up to double the speed of existing MoE frameworks.
The article details how the open‑source Flash‑MoE engine streams a 400‑billion‑parameter Mixture‑of‑Experts language model on an iPhone 17 Pro, achieving interactive‑level token throughput by eliminating Python dependencies, crafting a custom Metal pipeline, and streaming weights directly from SSD.
A Chinese Academy of Sciences team introduced the MatRIS‑MoE model and the Janus training framework, enabling a 11.5 billion‑parameter universal machine‑learning interatomic potential to be trained on two exascale systems at 1.2 EFLOPS, compressing weeks‑long training into a few hours.
AI for ScienceExascale trainingHigh-Performance Computing
The article details the Flash‑MoE project that streams the 400 billion‑parameter Qwen3.5‑397B‑A17B mixture‑of‑experts model on an iPhone 17 Pro, achieving up to 0.6 tokens per second with a custom Metal‑GPU pipeline, zero‑Python code, and SSD‑backed weight streaming that keeps only 5.5 GB in RAM.
Xiaomi has launched the MiMo 2.5 series, featuring a 1.02‑trillion‑parameter MoE model with 1 M‑token context, offers a token‑grant program for developers, and delivers benchmark scores that rival leading models such as DeepSeek‑V4‑Pro, Kimi K2, GPT‑5 and Gemini 3.0.
The article analyzes how both DeepSeek V4 and Meituan's LongCat‑2.0‑P preview, each with trillion‑scale parameters and 1 M‑token context, were trained and inferred entirely on Chinese‑made accelerators, detailing memory optimizations, deterministic operators, MoE redesigns, and massive multi‑card clusters that prove domestic compute can meet top‑tier AI workloads.
DeepSeek V4 introduces a suite of architectural innovations—including mixed attention, manifold‑constrained hyper‑connections, the Muon optimizer, and FP4‑aware quantization—that together slash million‑token inference cost to a tenth of its predecessor while delivering benchmark results that rival top‑tier closed‑source models.
Manifold AI's WorldScape 0.2 topped the WorldArena benchmark by excelling in visual quality, physics compliance, and 3D accuracy, while using only 10% of the parameters of competing models, thanks to a newly introduced MoE architecture that drives a new scaling law for world models.
DeepSeek V4, released in April 2026, introduces two versions—Pro and Flash—with up to 1.6 trillion parameters and a million‑token context window, leveraging hybrid attention, compressed KV cache, and specialized training techniques to dramatically cut hardware dependence and inference cost.
The article analyses DeepSeek V4’s architectural innovations—including Compressed Sparse Attention, Heavily Compressed Attention, a cross‑layer MoE design, and an Agent‑RL framework with Generative Reward Models and multi‑teacher distillation—while comparing its long‑context capabilities and efficiency to rival LLMs such as GLM, Kimi, Claude, GPT and Gemini.
Agent Reinforcement LearningCompressed Sparse AttentionGenerative Reward Model
DeepSeek released the V4 series—V4‑Pro (1.6 T total, 49 B active) and V4‑Flash (284 B total, 13 B active)—featuring three architectural upgrades, three inference modes, mixed‑precision FP4/FP8 weights, and benchmark results that place its programming ability at the top of open‑source models while supporting a million‑token context window.
DeepSeek V4 quietly debuted with a 1.6‑trillion‑parameter MoE model, introducing CSA+HCA compressed attention, mHC manifold‑constrained hyperconnections, and the Muon optimizer, achieving 1M‑token context at a quarter of V3’s cost, top Codeforces and LiveCodeBench scores, a 1/7 Opus price, MIT open‑source licensing, and dual‑stack Ascend NPU/NVIDIA GPU support.
DeepSeek V4, released on April 24 2026, offers two SKUs—Pro with 1.6 T total parameters and Flash with 284 B—both supporting a 1‑million‑token context window, ultra‑low inference cost, pricing as low as ¥0.2 per million tokens, Huawei Ascend deployment, and seamless OpenAI/Anthropic API compatibility.
Alibaba International Intelligent Technology showcases nine industrial‑grade recommendation and search papers accepted at SIGIR, WWW, and WSDM 2026, detailing sparse scaling, counterfactual multi‑task learning, generative recommendation, MoE routing, and multimodal semantic ID breakthroughs with extensive offline and online results.
The article analyzes Qwen3.6‑35B‑A3B’s MoE architecture, showing how its 30 B active parameters outperform larger dense models across programming, agent, and multimodal benchmarks, and examines the flagship Qwen3.6‑Max‑Preview’s substantial gains in world knowledge, instruction following, and third‑party rankings.
OpenMythos re‑creates the Claude Mythos architecture as a Recurrent‑Depth Transformer with MoE routing, achieving comparable performance to larger Transformers while using roughly half the parameters, and demonstrates systematic generalization and depth extrapolation through looped inference in latent space.
Qwen3.6‑35B‑A3B, a mixture‑of‑experts model that activates only 3 B parameters, outperforms leading AI systems across SWE‑bench, Terminal‑Bench, NL2Repo and several agentic coding benchmarks, while also achieving top scores in GPQA, HMMT and RealWorldQA, prompting a reassessment of domestic LLM capabilities.
DeepSeek's latest DeepGEMM update adds Mega MoE, a fused GPU kernel that collapses the entire Mixture‑of‑Experts pipeline and overlaps computation with NVLink communication, while also unveiling an FP4 indexer and FP8×FP4 precision experiments, signaling a push toward highly efficient large‑scale AI training.
Google DeepMind's Gemma 4 family introduces four open‑source models—including a 31B dense and a 26B MoE variant with 256K context—that deliver multimodal capabilities, tool‑use functions, and benchmark results rivaling much larger models while running on a single H100 GPU.
ProMoE introduces a two‑step routing MoE framework with explicit semantic guidance that tackles the high spatial redundancy and functional heterogeneity of visual tokens, enabling diffusion transformers to scale efficiently and outperform dense models and prior MoE approaches across generation, convergence, and scaling benchmarks.
By introducing a dynamic mixture‑of‑experts routing scheme and an end‑to‑end architecture that eliminates NMS and DFL, YOLO‑Master and YOLO26 dramatically cut compute waste and latency on edge devices, achieving up to 43% faster CPU inference while keeping model accuracy, with all code openly released.
Tencent’s YOLO-Master v2026.02 adds a Mixture‑of‑Experts architecture, zero‑overhead LoRA fine‑tuning, Sparse SAHI inference for large images, and Cluster‑Weighted NMS, delivering 3‑5× faster inference, up to 70% reduced training resources, and markedly higher detection accuracy across diverse benchmarks.
This article systematically dissects the engineering decisions behind frontier large‑language‑model training—covering architecture choices, attention variants, optimizer evolution, data‑curation strategies, scaling‑law insights, and post‑training SFT/RL pipelines—based on open‑source reports from seven leading AI laboratories.
The llmfit command‑line utility scans your CPU, RAM, GPU and VRAM, scores 157 models from over 30 providers, suggests the highest‑quality quantized version that fits, integrates with Ollama, and shows real‑world test results confirming its accuracy, though its model database is limited.
MiniMax’s new open‑source M2.5 model, built on a 230 billion‑parameter mixture‑of‑experts architecture that activates only 10 billion parameters per inference, delivers performance comparable to Claude Opus 4.6 across benchmarks, while costing roughly one‑tenth as much, and is already handling a large share of the company’s internal tasks.