BestHub
Sign in
Tagged articles

Mixture of Experts

179 articles · Page 1 of 2
Linyb Geek Road
Linyb Geek Road
Sep 29, 2026 · Artificial Intelligence

Reverse-Engineering Jev: 10K API Calls Expose Closed-Source Model Architecture

An independent researcher reverse-engineered TypeSafe's closed-source Jev classification model using 10,000 API calls, revealing its architecture uses a classification head with shared prefix inference, listwise scoring, and exceptional calibration (ECE 0.0313), challenging assumptions about black-box security.

JevMixture of Expertsblack-box probing
0 likes · 22 min read
Reverse-Engineering Jev: 10K API Calls Expose Closed-Source Model Architecture
Machine Heart
Machine Heart
Sep 24, 2026 · Artificial Intelligence

Edge AI Token Factories: Banma Smart's AutoOmni 2.0 Enables Deep AI Usage in Cars

At the 2026 Yunqi Conference, Banma Smart unveiled AutoOmni 2.0-23B-A3B, a sparse MoE edge model that achieves near-cloud performance on vehicle-grade chips, addressing memory, latency, and concurrency constraints while enabling privacy-preserving, low-cost AI with certified safety frameworks for mass production.

AI safetyASPICE CertificationAutoOmni
0 likes · 24 min read
Edge AI Token Factories: Banma Smart's AutoOmni 2.0 Enables Deep AI Usage in Cars
JavaGuide
JavaGuide
Sep 21, 2026 · Artificial Intelligence

Step 5 Preview Outperforms DeepSeek V4 Pro in Coding Tasks at Fraction of Opus 5 Cost

The article benchmarks StepFun's Step 5 Preview against DeepSeek V4 Pro across three coding challenges—2D animation, token bucket demo, and CSV analysis workbench—revealing Step 5 Preview delivers richer details and better test coverage despite longer generation times, with per-task cost at just 35% of GLM-5.3 and 12.5% of Claude Opus 5.

AI coding benchmarkCSV analysisDeepSeek V4 Pro
0 likes · 23 min read
Step 5 Preview Outperforms DeepSeek V4 Pro in Coding Tasks at Fraction of Opus 5 Cost
Machine Heart
Machine Heart
Sep 8, 2026 · Artificial Intelligence

SMELT: Looped Transformers Outperform Baselines Under Fair Budget Matching

The SMELT framework from Tsinghua and ByteDance Seed fairly compares Looped Transformers against baselines by matching compute, parameters, and KV cache, finding that looping the middle 50% of layers twice with a larger depth-width ratio consistently reduces validation loss across scales, saves 6.8–18% training compute, and yields downstream gains beyond loss reduction.

Budget MatchingLLM ArchitectureLooped Transformer
0 likes · 12 min read
SMELT: Looped Transformers Outperform Baselines Under Fair Budget Matching
Architects' Tech Alliance
Architects' Tech Alliance
Sep 6, 2026 · Artificial Intelligence

xDeepServe on CloudMatrix384: Full-Stack Design for Large-Scale MoE Model Serving

This article details Huawei's xDeepServe system for deploying massive Mixture-of-Experts models on the CloudMatrix384 supernode, covering the XCCL communication library, FlowServe decentralized serving engine, Transformerless execution architecture with prefill-decode and MoE-Attention decoupling, and hierarchical fault tolerance, achieving 2400 tokens/s per chip at 50ms TPOT.

Ascend 910CCloudMatrix384EPLB
0 likes · 18 min read
xDeepServe on CloudMatrix384: Full-Stack Design for Large-Scale MoE Model Serving
Xiaomi Tech
Xiaomi Tech
Sep 5, 2026 · Artificial Intelligence

Xiaomi-TabLDM: One Model for All Tabular Tasks via Synthetic Pretraining & Test-Time Scaling

Xiaomi releases Xiaomi-TabLDM, a foundation model for tabular data that uses large-scale synthetic pretraining, efficient model scaling with dual-stream feature groups and sparse MoE, and test-time scaling to achieve top-tier performance on four public benchmarks and real-world industrial tasks without per-dataset retraining.

BenchmarkMixture of ExpertsTest-Time Scaling
0 likes · 9 min read
Xiaomi-TabLDM: One Model for All Tabular Tasks via Synthetic Pretraining & Test-Time Scaling
Cambridge Mofang Notes
Cambridge Mofang Notes
Sep 1, 2026 · Artificial Intelligence

From Dense to MoE: Decoding Total vs. Activated Parameters

This article explains the distinction between total and activated parameters in Mixture-of-Experts (MoE) models, contrasting dense and sparse architectures, detailing expert routing mechanisms, and analyzing memory and compute implications across model loading, prefill, and decode stages.

MOEMixture of Expertsactivated parameters
0 likes · 15 min read
From Dense to MoE: Decoding Total vs. Activated Parameters
Tencent Technical Engineering
Tencent Technical Engineering
Aug 28, 2026 · Artificial Intelligence

Tencent Hunyuan Hy4 Preview: 770B Open-Source Model Claims Top Tier

Tencent releases Hunyuan Hy4 preview, a 770B parameter mixture-of-experts model with 49B activated parameters and 1M context length, achieving top-tier open-source performance across coding, office, gaming, and scientific tasks, with internal blind tests showing 2.99/4 score surpassing GLM 5.3 and Kimi K3, plus 31.8% inference throughput gains via self-optimization.

HunyuanHy4LLM
0 likes · 6 min read
Tencent Hunyuan Hy4 Preview: 770B Open-Source Model Claims Top Tier
SuanNi
SuanNi
Aug 21, 2026 · Artificial Intelligence

Ornith-1.5 Hits SOTA 9B/35B and Matches Claude Opus 4.8 at 397B

Ornith-1.5, an MIT‑licensed large‑model framework from DeepReinforce, introduces a self‑improving loop that autonomously generates tasks, builds scaffolds, and rolls out solutions, achieving state‑of‑the‑art performance at 9B and 35B scales and delivering benchmark scores comparable to Claude Opus 4.8 for its 397B MoE variant.

AIBenchmarkMixture of Experts
0 likes · 7 min read
Ornith-1.5 Hits SOTA 9B/35B and Matches Claude Opus 4.8 at 397B
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Aug 17, 2026 · Artificial Intelligence

N Load‑Balancing Techniques for Large‑Scale MoE Models

The article explains why token‑to‑expert routing in large Mixture‑of‑Experts models can become highly imbalanced, outlines the resulting compute bottlenecks, token overflow and expert degradation, and then surveys eight concrete load‑balancing strategies—ranging from importance‑based auxiliary losses to capacity‑aware token dropping and loss‑free bias control—detailing their mechanisms, trade‑offs, and practical considerations.

Auxiliary LossBalanced AssignmentCapacity Factor
0 likes · 17 min read
N Load‑Balancing Techniques for Large‑Scale MoE Models
TechVision Expert Circle
TechVision Expert Circle
Aug 17, 2026 · Artificial Intelligence

Open-Source LLMs Close the Gap: Low-Cost AI Poised to Redefine the Market

In early 2026, open-weight LLMs such as Llama 4, Qwen 3 and DeepSeek‑V3/R2 began matching or surpassing leading closed models like GPT‑4.5 and Claude Opus 5, driven by MoE architectures, FP8 precision, GRPO training and aggressive inference optimizations, prompting a reassessment of enterprise AI strategy.

AI economicsMixture of Expertsinference optimization
0 likes · 15 min read
Open-Source LLMs Close the Gap: Low-Cost AI Poised to Redefine the Market
Old Zhang's AI Learning
Old Zhang's AI Learning
Aug 17, 2026 · Artificial Intelligence

How Many GPUs Does Kimi K3 Need? Self‑Hosting vs API Cost Comparison

The article breaks down Kimi K3’s 2.8‑trillion‑parameter architecture, explains its 4‑bit MXFP4 quantization, calculates the ~1.4 TB memory requirement, shows that 8‑GPU clusters (e.g., NVIDIA B300 or AMD MI350X) are needed for self‑hosting, and compares these costs with the per‑token API pricing, highlighting when each option is economical.

API costDigitalOceanGPU requirements
0 likes · 14 min read
How Many GPUs Does Kimi K3 Need? Self‑Hosting vs API Cost Comparison
21CTO
21CTO
Aug 9, 2026 · Artificial Intelligence

ByteDance’s 10‑Trillion‑Parameter Gamble: Inside China’s Largest AI Model Pre‑training

ByteDance is reportedly pre‑training a 10‑trillion‑parameter AI model that dwarfs domestic rivals, uses a Mixture‑of‑Experts architecture with low activation ratios, demands roughly 36 000 Blackwell GPUs and $2.5 billion in hardware, and deliberately avoids distilling competitor models, raising questions about China’s AI frontier timeline.

AI scalingByteDanceDistillation avoidance
0 likes · 5 min read
ByteDance’s 10‑Trillion‑Parameter Gamble: Inside China’s Largest AI Model Pre‑training
21CTO
21CTO
Aug 8, 2026 · Artificial Intelligence

Jeff Dean Discusses the Next Decade of AI Just 12 Hours After Leaving Google

In a Stanford interview hosted by Dawn Song, Jeff Dean reflects on the origins of MoE, the transformative impact of deep learning, lessons from TensorFlow, how to spot breakthrough directions, AI agent risks, and his new venture Discovery Loop that aims to automate scientific research.

AI safetyArtificial IntelligenceDiscovery Loop
0 likes · 12 min read
Jeff Dean Discusses the Next Decade of AI Just 12 Hours After Leaving Google
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Aug 5, 2026 · Artificial Intelligence

Inside K3: How Stable Latent MoE and MLA Attention Are Designed

The article examines K3’s architecture—combining KDA, MLA, Stable Latent MoE and AttnRes—detailing the replacement of SwiGLU with SiTU‑GLU, the addition of RMSNorm for training stability, the Quantile Balancing load‑balancing scheme, and the trade‑offs behind its MLA and NoPE attention choices.

K3MLA AttentionMixture of Experts
0 likes · 15 min read
Inside K3: How Stable Latent MoE and MLA Attention Are Designed
21CTO
21CTO
Aug 3, 2026 · Artificial Intelligence

JetBrains’ Open‑Source 12B Code Model Mellum2: Private Deployment and High‑Throughput Over Claude Code

JetBrains released the open‑source 12‑billion‑parameter Mellum2 model, a MoE‑based code AI that delivers private on‑prem deployment, high‑throughput inference, and strong code‑generation benchmarks, positioning it as a fast, specialized alternative to Claude Code and other proprietary models.

BenchmarkMellum2Mixture of Experts
0 likes · 7 min read
JetBrains’ Open‑Source 12B Code Model Mellum2: Private Deployment and High‑Throughput Over Claude Code
Black & White Path
Black & White Path
Aug 2, 2026 · Artificial Intelligence

Running a 2.8‑Trillion‑Parameter K3 Model on 4 GB VRAM with AirLLM

AirLLM introduces layer‑wise inference and per‑expert streaming to decouple VRAM usage from model size, enabling the 2.8‑trillion‑parameter Kimi K3 LLM to run on a single consumer‑grade GPU while preserving full‑precision accuracy and offering security‑focused insights.

AirLLMKimi K3Layer-wise Inference
0 likes · 9 min read
Running a 2.8‑Trillion‑Parameter K3 Model on 4 GB VRAM with AirLLM
ITPUB
ITPUB
Jul 28, 2026 · Artificial Intelligence

Why Kimi K3’s Open‑Source Release Puts China at the Forefront of Global AI

Kimi K3, a 2.8‑trillion‑parameter MoE model with a 100 k‑token context, has been fully open‑sourced along with its weights, technical report and infra (MoonEP, FlashKDA, AgentEnv), delivering programming and agent benchmark results that rival top closed models such as Claude Fable 5 and GPT‑5.6 while sparking debate over alleged distillation and emphasizing AI safety and open‑weight governance.

AI benchmarksAI safetyKimi K3
0 likes · 11 min read
Why Kimi K3’s Open‑Source Release Puts China at the Forefront of Global AI
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Jul 27, 2026 · Artificial Intelligence

Scaling Residual Streams Efficiently: From DeepSeek mHC to xHC’s 16‑Stream Expansion

The blog details how xHC expands language‑model residual streams to 16, achieving nearly double the gain of DeepSeek mHC on 18B and 28B MoE models, and explains the design of Temporal Feature Augmentation and Sparse Write that make large‑N scaling both effective and affordable.

Hyper-ConnectionsMixture of Expertslarge language models
0 likes · 24 min read
Scaling Residual Streams Efficiently: From DeepSeek mHC to xHC’s 16‑Stream Expansion
AI Programming Lab
AI Programming Lab
Jul 21, 2026 · Artificial Intelligence

How Kimi K3 and Qwen3.8‑Max Reach 2T+ Parameters: The Evolution of Chinese LLM Architecture

The article examines how Chinese LLMs such as Kimi K3 (2.8 T) and Qwen 3.8‑Max (2.4 T) achieved 2‑trillion‑parameter scales by progressively decoupling parameters from compute with Mixture‑of‑Experts, optimizing attention, and introducing training tricks like Muon and low‑precision quantization, tracing four years of architectural advances.

Mixture of Expertsattention mechanismslarge language models
0 likes · 11 min read
How Kimi K3 and Qwen3.8‑Max Reach 2T+ Parameters: The Evolution of Chinese LLM Architecture
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Jul 19, 2026 · Artificial Intelligence

How Kimi K3 Highlights Latent MoE as the Next Turning Point in Mixture‑of‑Experts Architecture

Latent MoE, demonstrated by NVIDIA’s Nemotron 3 Super and Moonshot AI’s 2.8 T‑parameter Kimi K3, compresses expert computations into a lower‑dimensional latent space, cutting memory reads and All‑to‑All traffic by fourfold, enabling more experts per token, higher accuracy, and up to 3.5× faster inference.

AI model scalingKimi K3Latent MoE
0 likes · 10 min read
How Kimi K3 Highlights Latent MoE as the Next Turning Point in Mixture‑of‑Experts Architecture
21CTO
21CTO
Jul 17, 2026 · Artificial Intelligence

Kimi K3 Unveiled: 2.8 Trillion‑Parameter Open‑Source LLM Sets New Record

On July 16, the Moon‑of‑Darkness team released Kimi K3, a 2.8‑trillion‑parameter open‑source large language model that introduces mixed‑linear attention, attention residuals, and a highly efficient Mixture‑of‑Experts design, achieving roughly 2.5× the scaling efficiency of its predecessor while approaching the performance of top closed‑source models.

Kimi K3Mixture of Expertslarge language model
0 likes · 6 min read
Kimi K3 Unveiled: 2.8 Trillion‑Parameter Open‑Source LLM Sets New Record
SuanNi
SuanNi
Jul 17, 2026 · Artificial Intelligence

Kimi K3: The World’s First 3‑Trillion‑Parameter Open‑Source Model

Kimi K3, a 2.8‑trillion‑parameter open‑source LLM, outperforms top closed‑source models in benchmarks, excels at long‑range coding, GPU kernel optimization, and multimodal tasks, while introducing novel attention mechanisms, a compact Triton‑like compiler, and even a prototype ASIC chip.

BenchmarkGPU compilationKimi K3
0 likes · 9 min read
Kimi K3: The World’s First 3‑Trillion‑Parameter Open‑Source Model
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Jul 16, 2026 · Artificial Intelligence

Inkling – A 975‑Billion‑Parameter Open‑Weight Multimodal Model from Thinking Machines

Thinking Machines Lab unveiled Inkling, a 975‑billion‑parameter open‑weight multimodal model featuring a hybrid‑expert Transformer, 1‑million‑token context, and extensive benchmark results, alongside the lighter Inkling‑Small, with detailed architecture, training methodology, reinforcement‑learning enhancements, and practical examples of web‑app generation and tool‑calling.

BenchmarkInklingMixture of Experts
0 likes · 15 min read
Inkling – A 975‑Billion‑Parameter Open‑Weight Multimodal Model from Thinking Machines
TechVision Expert Circle
TechVision Expert Circle
Jul 16, 2026 · Artificial Intelligence

Enterprise AI Trends for H2 2026: Key Priorities for Tech Leaders

In the second half of 2026, enterprise AI shifts from adoption to reliable, cost‑effective deployment, with six key trends—including multi‑agent orchestration, GraphRAG retrieval, MoE model clusters, AI observability, built‑in data governance, and reorganized AI engineering roles—guiding tech leaders toward trustworthy AI systems.

AI AgentAI ObservabilityAI Team Structure
0 likes · 13 min read
Enterprise AI Trends for H2 2026: Key Priorities for Tech Leaders
Architecture Digest
Architecture Digest
Jul 14, 2026 · Artificial Intelligence

How DeepSeek V4’s CSA+HCA Architecture Turns Million‑Token Context into a Cheap Feature

DeepSeek V4 introduces a hybrid CSA/HCA attention mechanism and a Mixture‑of‑Experts design that reduces inference cost to 27% of its predecessor, enables a 1‑million‑token context as a standard feature, and runs efficiently on both NVIDIA GPUs and Huawei Ascend NPU, dramatically lowering the barrier for Java developers to integrate large‑scale AI.

CSADeepSeek-V4HCA
0 likes · 10 min read
How DeepSeek V4’s CSA+HCA Architecture Turns Million‑Token Context into a Cheap Feature
Black & White Path
Black & White Path
Jul 14, 2026 · Artificial Intelligence

SuperGemma 26B: The Fully Uncensored ‘Zero‑Guardrails’ AI Model Explained

Independent developer David Ondrej released SuperGemma 26B, an uncensored fork of Google’s Gemma 4 26B that removes all safety guardrails, runs locally on consumer‑grade GPUs, and has sparked intense debate over its technical merits, deployment simplicity, and the security risks of a truly unrestricted AI model.

AI safetyGemma 4Mixture of Experts
0 likes · 11 min read
SuperGemma 26B: The Fully Uncensored ‘Zero‑Guardrails’ AI Model Explained
Geek Labs
Geek Labs
Jul 14, 2026 · Artificial Intelligence

How Colibri Runs a 744 B MoE Model on CPU with Only 25 GB RAM

Colibri demonstrates that the 744‑billion‑parameter GLM‑5.2 MoE model can be executed on a pure‑CPU system using roughly 25 GB of memory by streaming expert weights from disk, int4‑quantizing dense parameters, and employing a compact C inference engine.

CPU inferenceGLM-5.2MLA Attention
0 likes · 7 min read
How Colibri Runs a 744 B MoE Model on CPU with Only 25 GB RAM
Machine Heart
Machine Heart
Jul 13, 2026 · Artificial Intelligence

Breaking the Forgetting Barrier: CaRE Scales Continual Learning to 300+ Tasks

The paper introduces CaRE, a scalable continual‑learning framework that leverages a bi‑level routing mixture‑of‑experts to successfully train Vision Transformers on over 300 non‑overlapping tasks, outperforming existing baselines and accompanied by a new 1,000‑class benchmark, OmniBenchmark‑1K.

Bi-Level RoutingCaREContinual Learning
0 likes · 11 min read
Breaking the Forgetting Barrier: CaRE Scales Continual Learning to 300+ Tasks
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Jul 10, 2026 · Artificial Intelligence

World’s First Embodied‑Native Action Model: Inside LingBot‑VA 2.0

LingBot‑VA 2.0 introduces the industry’s first embodied‑native pre‑trained robot brain, combining causal action modeling, a sparse MoE backbone, a semantic VAE tokenizer and asynchronous foresight reasoning to achieve six‑fold inference speedup, single‑GPU deployment and a 93.6% success rate on the RoboTwin 2.0 benchmark.

Causal ModelingEmbodied AIForesight Reasoning
0 likes · 11 min read
World’s First Embodied‑Native Action Model: Inside LingBot‑VA 2.0
Machine Heart
Machine Heart
Jul 9, 2026 · Artificial Intelligence

LingBot-Video: The Open‑Source Video Foundation Model Tailored for Robots

LingBot-Video is an open‑source video‑generation foundation model built for embodied AI, combining a sparse‑Mixture‑of‑Experts architecture, a multi‑stage data curriculum and six‑dimensional reward learning to achieve physically consistent video synthesis that outperforms existing open‑source baselines in both visual quality and robotic relevance.

Embodied AIMixture of ExpertsMultimodal Model
0 likes · 22 min read
LingBot-Video: The Open‑Source Video Foundation Model Tailored for Robots
Machine Heart
Machine Heart
Jul 8, 2026 · Artificial Intelligence

How LingBot‑VLA 2.0 Powers 20 Robot Configurations with an Open‑Source Embodied Brain

LingBot‑VLA 2.0 introduces a token‑level loss‑free MoE, dual‑query distillation, and a 60k‑hour heterogeneous dataset to achieve cross‑embodiment visual‑language‑action capabilities across 20 robot morphologies, delivering superior benchmark performance and sub‑130 ms inference while being fully open‑sourced.

Embodied AIFuture PredictionMixture of Experts
0 likes · 16 min read
How LingBot‑VLA 2.0 Powers 20 Robot Configurations with an Open‑Source Embodied Brain
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Jul 1, 2026 · Artificial Intelligence

SAME: Stabilizing MoE to Reduce Dual Forgetting in Multimodal Continual Instruction Tuning

The paper identifies routing drift and expert drift as the two main causes of forgetting in multimodal continual instruction tuning (MCIT) and proposes SAME, which combines spectral‑aware routing, curvature‑aware scaling, and adaptive expert activation to keep MoE models stable, efficient, and less forgetful across long task sequences.

Continual LearningICML 2026Mixture of Experts
0 likes · 19 min read
SAME: Stabilizing MoE to Reduce Dual Forgetting in Multimodal Continual Instruction Tuning
DeepHub IMBA
DeepHub IMBA
Jun 22, 2026 · Artificial Intelligence

How DiffusionGemma Shifts LLM Inference Bottleneck from Memory Bandwidth to Compute

DiffusionGemma, an experimental discrete text diffusion model built on the 26B MoE Gemma‑4 architecture, generates whole 256‑token blocks with bidirectional attention, moving the inference bottleneck from memory bandwidth to GPU compute, achieving up to four‑fold speed gains on H100 and RTX 5090 GPUs, though with lower output quality than standard autoregressive models.

DiffusionGemmaGPU performanceLLM inference
0 likes · 7 min read
How DiffusionGemma Shifts LLM Inference Bottleneck from Memory Bandwidth to Compute
AI Engineer Programming
AI Engineer Programming
Jun 17, 2026 · Artificial Intelligence

Local LLMs Viable: Sparse Attention, MoE, KV Compression, Multi‑Token Prediction

In early 2026, open‑source local large language models become practical alternatives thanks to sparse attention, MoE routing, latent KV compression, multi‑token prediction, and 4‑bit quantization, while hardware memory shortages and benchmark gaps with closed‑source models shape their deployment choices.

4-bit quantizationKV compressionLocal LLM
0 likes · 13 min read
Local LLMs Viable: Sparse Attention, MoE, KV Compression, Multi‑Token Prediction
AI Insight Log
AI Insight Log
Jun 12, 2026 · Artificial Intelligence

Kimi K2.7 Code: 1T MoE Model Cuts Tokens 30% and Beats Claude Opus on MCP Calls

The newly released Kimi K2.7 Code, a 1‑trillion‑parameter mixture‑of‑experts model that activates only 32 B parameters per inference, offers a 256 K context window, supports multimodal input, improves benchmark scores by up to 31.5 % over K2.6, reduces inference token usage by about 30 %, and achieves an 81.1 MCP tool‑call score surpassing Claude Opus 4.8, while providing a CLI installation command and usage guidelines.

BenchmarkCoding ModelKimi
0 likes · 7 min read
Kimi K2.7 Code: 1T MoE Model Cuts Tokens 30% and Beats Claude Opus on MCP Calls
HyperAI Super Neural
HyperAI Super Neural
Jun 12, 2026 · Artificial Intelligence

DiffusionGemma Boosts Text Generation Speed Up to 4× with Discrete Diffusion

Google’s open‑source DiffusionGemma model leverages a 26‑billion‑parameter Mixture‑of‑Experts architecture and discrete diffusion decoding to generate whole text blocks, achieving up to four times faster generation—over 1100 tokens/s on an NVIDIA H100 and 700 tokens/s on an RTX 5090—while activating only 3.8 billion parameters during inference.

DiffusionGemmaDiscrete DiffusionGPU acceleration
0 likes · 4 min read
DiffusionGemma Boosts Text Generation Speed Up to 4× with Discrete Diffusion
Machine Heart
Machine Heart
Jun 11, 2026 · Artificial Intelligence

Google Releases DiffusionGemma 26B MoE—Text Generation Up to 4× Faster

DiffusionGemma, Google's new 26‑billion‑parameter Mixture‑of‑Experts model, replaces token‑by‑token autoregression with a diffusion‑style output head that generates whole text blocks, delivering up to four‑fold speed gains on consumer GPUs while offering bidirectional attention and self‑correction, albeit with lower quality than standard Gemma 4.

DiffusionGemmaGPU accelerationMixture of Experts
0 likes · 6 min read
Google Releases DiffusionGemma 26B MoE—Text Generation Up to 4× Faster
Data Party THU
Data Party THU
Jun 6, 2026 · Artificial Intelligence

How a 400B MoE Model Runs on iPhone 17 Pro with Flash‑MoE

The article details how the open‑source Flash‑MoE engine enables the 400B‑parameter Qwen3.5‑397B‑A17B mixture‑of‑experts model to run on an iPhone 17 Pro, achieving about 0.6 tokens per second through a custom Metal pipeline, GCD‑driven SSD streaming, and aggressive caching strategies.

400BFlash-MoELLM inference
0 likes · 6 min read
How a 400B MoE Model Runs on iPhone 17 Pro with Flash‑MoE
Data Party THU
Data Party THU
May 29, 2026 · Artificial Intelligence

Token Superposition Training: 2.5× Faster LLM Pre‑training Without Model Changes

The article presents Token Superposition Training (TST), which temporarily averages embeddings of non‑overlapping token bags and predicts groups of tokens in a first phase before reverting to standard token‑wise prediction, achieving up to 2.5× pre‑training speedup on 10B‑1B MoE models without altering model architecture or inference.

LLM pretrainingMCE lossMixture of Experts
0 likes · 9 min read
Token Superposition Training: 2.5× Faster LLM Pre‑training Without Model Changes
Architect's Guide
Architect's Guide
May 29, 2026 · Artificial Intelligence

What Makes DeepSeek V4 Different? A Deep Technical Dive into Its Innovations

DeepSeek V4 introduces a suite of architectural breakthroughs—including mixed‑expert MoE, manifold‑constrained hyper‑connections, CSA/HCA hybrid attention, and FP4 quantization—that slash inference cost by up to tenfold while delivering million‑token context, competitive benchmarks, dual model variants, and a disruptive pricing strategy.

AI model benchmarkAgentic AIDeepSeek-V4
0 likes · 41 min read
What Makes DeepSeek V4 Different? A Deep Technical Dive into Its Innovations
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
May 22, 2026 · Artificial Intelligence

20‑Year‑Old Transformer Co‑author Open‑Sources a 218‑Billion‑Parameter Model

Cohere’s Command A+ model, built by Transformer co‑author Aidan Gomez and backed by Nick Frosst, packs 218 billion parameters but activates only 25 billion at inference, uses a lossless 4‑bit quantization scheme, offers native citation support, runs on a single B200 or two H100 GPUs, and is released under an Apache 2.0 license, marking a major shift toward truly open‑source, enterprise‑ready large language models.

AIApache-2.0Cohere
0 likes · 12 min read
20‑Year‑Old Transformer Co‑author Open‑Sources a 218‑Billion‑Parameter Model
Machine Heart
Machine Heart
May 22, 2026 · Artificial Intelligence

Breaking the Echo Chamber: MP‑MoE Introduces Ensemble‑Pruning for Diverse Experts

The paper presents MP‑MoE, a new Mixture‑of‑Experts architecture that replaces top‑k routing with Mahalanobis‑based ensemble pruning, explicitly encouraging expert diversity via a co‑occurrence matrix, and uses an efficient greedy algorithm with incremental Cholesky updates, achieving higher performance with minimal training overhead and no inference cost.

Expert DiversityICML 2026MP-MoE
0 likes · 8 min read
Breaking the Echo Chamber: MP‑MoE Introduces Ensemble‑Pruning for Diverse Experts
Alibaba International Intelligent Technology
Alibaba International Intelligent Technology
May 22, 2026 · Artificial Intelligence

How SORT Transforms Precision Ranking with a Transformer‑Based Architecture

SORT re‑architects industrial‑scale ranking by shifting to a request‑centric data paradigm, integrating sparse and MoE optimizations into a Transformer backbone, and delivering significant CTR‑AUC, FLOPs, and online metric improvements while maintaining high training and inference efficiency.

Large Model ScalingMixture of ExpertsTransformer
0 likes · 19 min read
How SORT Transforms Precision Ranking with a Transformer‑Based Architecture
Machine Heart
Machine Heart
May 20, 2026 · Artificial Intelligence

Can Tabular Anomaly Detection Move Beyond One‑for‑One? OFA‑TAD Introduces a One‑for‑All Paradigm

Tabular anomaly detection traditionally requires training a separate model for each dataset (one‑for‑one), but the new OFA‑TAD framework trains once on multiple source tables and directly transfers to unseen target tables without fine‑tuning, leveraging multi‑view distance encoding, MoE fusion, and synthetic pseudo‑anomalies to achieve state‑of‑the‑art performance across 34 datasets in 14 domains.

Mixture of ExpertsOFA-TADmulti-view distance
0 likes · 10 min read
Can Tabular Anomaly Detection Move Beyond One‑for‑One? OFA‑TAD Introduces a One‑for‑All Paradigm
Data Party THU
Data Party THU
May 17, 2026 · Artificial Intelligence

How DeepSeek Leverages MoE Parallelism: GPU Compute and Communication Optimizations

The article dissects DeepSeek's MoE model‑parallel strategy, explaining how GPU compute and communication are overlapped through expert, pipeline, and ZeRO‑1 parallelism, and introduces DualPipe and Waved‑EP kernels that enable efficient training on large‑scale hardware.

DeepSeekGPU Communication OverlapMixture of Experts
0 likes · 18 min read
How DeepSeek Leverages MoE Parallelism: GPU Compute and Communication Optimizations
PaperAgent
PaperAgent
May 13, 2026 · Artificial Intelligence

One-for-All Multi-Agent Collaboration: Adaptive Cross-Task Topology Design

The paper introduces OFA-MAS, a one‑for‑all multi‑agent system that learns a universal topology designer using task‑aware graph encoding and a Mixture‑of‑Experts generator, achieving superior performance, OOD generalization, robustness, and efficiency across six major benchmarks.

LLMMixture of ExpertsTask-Aware Graph Encoder
0 likes · 14 min read
One-for-All Multi-Agent Collaboration: Adaptive Cross-Task Topology Design
Lao Guo's Learning Space
Lao Guo's Learning Space
May 12, 2026 · Artificial Intelligence

Demystifying the Core Technologies Behind ChatGPT, GPT‑4, and DeepSeek

This article breaks down the key algorithms that power large‑language models—Transformer, Mixture‑of‑Experts, Flash Attention, KV‑Cache, Multi‑Token Prediction, quantization, Chain‑of‑Thought and Retrieval‑Augmented Generation—explaining how each contributes to the performance of ChatGPT, GPT‑4 and DeepSeek.

Flash AttentionKV CacheMixture of Experts
0 likes · 10 min read
Demystifying the Core Technologies Behind ChatGPT, GPT‑4, and DeepSeek
Old Zhang's AI Learning
Old Zhang's AI Learning
May 11, 2026 · Artificial Intelligence

Open‑Source Qwen3.6‑35B‑A3B Runs at 162 tok/s on a Single RTX 5090

The article introduces the open‑source Qwen3.6‑35B‑A3B model, explains its MoE architecture, three‑stage LoRA fine‑tuning, shows benchmark results where it achieves 161.9 tok/s on an RTX 5090—2.6× faster than a dense 27B counterpart—and discusses deployment tips, quantized GGUF release, and known compatibility pitfalls.

GGUF quantizationLoRA fine-tuningMixture of Experts
0 likes · 7 min read
Open‑Source Qwen3.6‑35B‑A3B Runs at 162 tok/s on a Single RTX 5090
Old Zhang's AI Learning
Old Zhang's AI Learning
May 7, 2026 · Artificial Intelligence

How Unsloth and NVIDIA Boost Consumer‑GPU LLM Training by ~25% with Three Simple Optimizations

Unsloth and NVIDIA identified three low‑level bottlenecks in LLM fine‑tuning on consumer GPUs—repeated packed‑sequence metadata construction, serialized copy‑and‑compute during gradient checkpointing, and per‑expert routing overhead in MoE—and applied targeted patches that together deliver roughly a 25% speedup without changing hardware, code, or frameworks.

GPU optimizationGradient CheckpointingLLM training
0 likes · 12 min read
How Unsloth and NVIDIA Boost Consumer‑GPU LLM Training by ~25% with Three Simple Optimizations
AI Engineer Programming
AI Engineer Programming
May 7, 2026 · Artificial Intelligence

How Cursor Turned Its Coding Agent from Demo to Production

The article examines Cursor's journey of shipping its Composer coding agent, detailing the agentic AI model, system architecture, and the three major production challenges—diff handling, latency accumulation, and sandbox scaling—along with the engineering solutions that enabled reliable, fast, and adoptable AI‑driven code generation.

Agentic AICursorMixture of Experts
0 likes · 16 min read
How Cursor Turned Its Coding Agent from Demo to Production
Machine Heart
Machine Heart
May 4, 2026 · Artificial Intelligence

Mega MoE vs SonicMoE: Which Will Lead the Next AI Speed Race?

SonicMoE, a new ultra‑fast Mixture‑of‑Experts model from Tri Dao and Ion Stoica’s team, achieves peak throughput on Nvidia Blackwell GPUs, outperforms DeepSeek’s DeepGEMM, and introduces algorithmic redesigns that decouple activation memory from expert granularity while fusing I/O‑aware kernels for up to double the speed of existing MoE frameworks.

AI performanceBlackwellGPU acceleration
0 likes · 12 min read
Mega MoE vs SonicMoE: Which Will Lead the Next AI Speed Race?
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
May 3, 2026 · Artificial Intelligence

Running a 400B Mixture‑of‑Experts LLM on iPhone 17 Pro: Inside Flash‑MoE

The article details how the open‑source Flash‑MoE engine streams a 400‑billion‑parameter Mixture‑of‑Experts language model on an iPhone 17 Pro, achieving interactive‑level token throughput by eliminating Python dependencies, crafting a custom Metal pipeline, and streaming weights directly from SSD.

Apple SiliconFlash-MoEGCD
0 likes · 7 min read
Running a 400B Mixture‑of‑Experts LLM on iPhone 17 Pro: Inside Flash‑MoE
Data Party THU
Data Party THU
May 2, 2026 · Artificial Intelligence

Training an 11.5 B‑parameter Universal Interatomic Potential in Hours on Exascale Supercomputers

A Chinese Academy of Sciences team introduced the MatRIS‑MoE model and the Janus training framework, enabling a 11.5 billion‑parameter universal machine‑learning interatomic potential to be trained on two exascale systems at 1.2 EFLOPS, compressing weeks‑long training into a few hours.

AI for ScienceExascale trainingML interatomic potentials
0 likes · 8 min read
Training an 11.5 B‑parameter Universal Interatomic Potential in Hours on Exascale Supercomputers
Machine Heart
Machine Heart
May 1, 2026 · Artificial Intelligence

How a 400B Mixture‑of‑Experts Model Runs on the iPhone 17 Pro

The article details the Flash‑MoE project that streams the 400 billion‑parameter Qwen3.5‑397B‑A17B mixture‑of‑experts model on an iPhone 17 Pro, achieving up to 0.6 tokens per second with a custom Metal‑GPU pipeline, zero‑Python code, and SSD‑backed weight streaming that keeps only 5.5 GB in RAM.

Flash-MoELLMMetal
0 likes · 7 min read
How a 400B Mixture‑of‑Experts Model Runs on the iPhone 17 Pro
Machine Heart
Machine Heart
Apr 30, 2026 · Artificial Intelligence

Beyond DeepSeek V4: A Trillion‑Parameter LLM Trained End‑to‑End on Domestic Chips

The article analyzes how both DeepSeek V4 and Meituan's LongCat‑2.0‑P preview, each with trillion‑scale parameters and 1 M‑token context, were trained and inferred entirely on Chinese‑made accelerators, detailing memory optimizations, deterministic operators, MoE redesigns, and massive multi‑card clusters that prove domestic compute can meet top‑tier AI workloads.

Deterministic OpsDomestic AI ChipLongCat
0 likes · 13 min read
Beyond DeepSeek V4: A Trillion‑Parameter LLM Trained End‑to‑End on Domestic Chips
Architect's Must-Have
Architect's Must-Have
Apr 28, 2026 · Artificial Intelligence

Why DeepSeek V4 Stands Apart: A Deep Dive into Its Architecture and Performance

DeepSeek V4 introduces a suite of architectural innovations—including mixed attention, manifold‑constrained hyper‑connections, the Muon optimizer, and FP4‑aware quantization—that together slash million‑token inference cost to a tenth of its predecessor while delivering benchmark results that rival top‑tier closed‑source models.

BenchmarkDeepSeek-V4FP4 quantization
0 likes · 44 min read
Why DeepSeek V4 Stands Apart: A Deep Dive into Its Architecture and Performance
CodeTrend
CodeTrend
Apr 26, 2026 · Artificial Intelligence

DeepSeek V4 Architecture: High‑Efficiency Long‑Context Model Design

DeepSeek V4, released in April 2026, introduces two versions—Pro and Flash—with up to 1.6 trillion parameters and a million‑token context window, leveraging hybrid attention, compressed KV cache, and specialized training techniques to dramatically cut hardware dependence and inference cost.

DeepSeekFP4Mixture of Experts
0 likes · 5 min read
DeepSeek V4 Architecture: High‑Efficiency Long‑Context Model Design
AI2ML AI to Machine Learning
AI2ML AI to Machine Learning
Apr 25, 2026 · Artificial Intelligence

How DeepSeek V4 Advances Structured Optimization in the Large‑Model Era

The article analyses DeepSeek V4’s architectural innovations—including Compressed Sparse Attention, Heavily Compressed Attention, a cross‑layer MoE design, and an Agent‑RL framework with Generative Reward Models and multi‑teacher distillation—while comparing its long‑context capabilities and efficiency to rival LLMs such as GLM, Kimi, Claude, GPT and Gemini.

Agent Reinforcement LearningCompressed Sparse AttentionDeepSeek-V4
0 likes · 7 min read
How DeepSeek V4 Advances Structured Optimization in the Large‑Model Era
Architect's Tech Stack
Architect's Tech Stack
Apr 25, 2026 · Artificial Intelligence

DeepSeek‑V4 Launch: 1.6 T Parameters, 1 M‑Token Context, Programming Skills Lead Open‑Source Rankings

DeepSeek released the V4 series—V4‑Pro (1.6 T total, 49 B active) and V4‑Flash (284 B total, 13 B active)—featuring three architectural upgrades, three inference modes, mixed‑precision FP4/FP8 weights, and benchmark results that place its programming ability at the top of open‑source models while supporting a million‑token context window.

AI architectureBenchmarkDeepSeek
0 likes · 5 min read
DeepSeek‑V4 Launch: 1.6 T Parameters, 1 M‑Token Context, Programming Skills Lead Open‑Source Rankings
ArcThink
ArcThink
Apr 25, 2026 · Artificial Intelligence

DeepSeek V4’s Silent Launch: 1.6 T Parameters, Triple Innovation, and Redefined Accessibility

DeepSeek V4 quietly debuted with a 1.6‑trillion‑parameter MoE model, introducing CSA+HCA compressed attention, mHC manifold‑constrained hyperconnections, and the Muon optimizer, achieving 1M‑token context at a quarter of V3’s cost, top Codeforces and LiveCodeBench scores, a 1/7 Opus price, MIT open‑source licensing, and dual‑stack Ascend NPU/NVIDIA GPU support.

BenchmarkDeepSeek-V4Manifold-constrained Hyperconnection
0 likes · 17 min read
DeepSeek V4’s Silent Launch: 1.6 T Parameters, Triple Innovation, and Redefined Accessibility
AI Agent Super App
AI Agent Super App
Apr 24, 2026 · Artificial Intelligence

DeepSeek V4 Launches with 1.6 T Parameters and 1 Million‑Token Context

DeepSeek V4, released on April 24 2026, offers two SKUs—Pro with 1.6 T total parameters and Flash with 284 B—both supporting a 1‑million‑token context window, ultra‑low inference cost, pricing as low as ¥0.2 per million tokens, Huawei Ascend deployment, and seamless OpenAI/Anthropic API compatibility.

AI pricingAPI CompatibilityDeepSeek
0 likes · 7 min read
DeepSeek V4 Launches with 1.6 T Parameters and 1 Million‑Token Context
Alibaba International Intelligent Technology
Alibaba International Intelligent Technology
Apr 24, 2026 · Artificial Intelligence

Alibaba International AI Team Lands Multiple Papers at SIGIR, WWW, and WSDM 2026

Alibaba International Intelligent Technology showcases nine industrial‑grade recommendation and search papers accepted at SIGIR, WWW, and WSDM 2026, detailing sparse scaling, counterfactual multi‑task learning, generative recommendation, MoE routing, and multimodal semantic ID breakthroughs with extensive offline and online results.

Mixture of Expertscounterfactual learninge‑commerce AI
0 likes · 24 min read
Alibaba International AI Team Lands Multiple Papers at SIGIR, WWW, and WSDM 2026
SuanNi
SuanNi
Apr 21, 2026 · Artificial Intelligence

How Qwen3.6‑35B‑A3B Matches Dense Models with Only 30 B Active Parameters

The article analyzes Qwen3.6‑35B‑A3B’s MoE architecture, showing how its 30 B active parameters outperform larger dense models across programming, agent, and multimodal benchmarks, and examines the flagship Qwen3.6‑Max‑Preview’s substantial gains in world knowledge, instruction following, and third‑party rankings.

AI evaluationBenchmarkMixture of Experts
0 likes · 5 min read
How Qwen3.6‑35B‑A3B Matches Dense Models with Only 30 B Active Parameters
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Apr 21, 2026 · Artificial Intelligence

How a 22‑Year‑Old Reversed‑Engineered Mythos into OpenMythos Using MoE and DeepSeek‑Inspired Attention

OpenMythos re‑creates the Claude Mythos architecture as a Recurrent‑Depth Transformer with MoE routing, achieving comparable performance to larger Transformers while using roughly half the parameters, and demonstrates systematic generalization and depth extrapolation through looped inference in latent space.

AI architectureLooped Language ModelsMixture of Experts
0 likes · 6 min read
How a 22‑Year‑Old Reversed‑Engineered Mythos into OpenMythos Using MoE and DeepSeek‑Inspired Attention
AI Large-Model Wave and Transformation Guide
AI Large-Model Wave and Transformation Guide
Apr 18, 2026 · Artificial Intelligence

Does Qwen3.6‑35B‑A3B Really Outclass All AI Coding Models? Inside the Benchmark Breakdown

Qwen3.6‑35B‑A3B, a mixture‑of‑experts model that activates only 3 B parameters, outperforms leading AI systems across SWE‑bench, Terminal‑Bench, NL2Repo and several agentic coding benchmarks, while also achieving top scores in GPQA, HMMT and RealWorldQA, prompting a reassessment of domestic LLM capabilities.

AI codingBenchmarkChinese AI
0 likes · 7 min read
Does Qwen3.6‑35B‑A3B Really Outclass All AI Coding Models? Inside the Benchmark Breakdown
Machine Heart
Machine Heart
Apr 17, 2026 · Artificial Intelligence

DeepSeek Introduces Mega MoE and FP4 Indexer – Inside the New GPU Fusion Kernel

DeepSeek's latest DeepGEMM update adds Mega MoE, a fused GPU kernel that collapses the entire Mixture‑of‑Experts pipeline and overlaps computation with NVLink communication, while also unveiling an FP4 indexer and FP8×FP4 precision experiments, signaling a push toward highly efficient large‑scale AI training.

DeepGEMMDeepSeekFP4 Indexer
0 likes · 5 min read
DeepSeek Introduces Mega MoE and FP4 Indexer – Inside the New GPU Fusion Kernel
Machine Heart
Machine Heart
Mar 31, 2026 · Artificial Intelligence

ProMoE: Explicit Routing Breaks the Scaling Bottleneck of Diffusion‑Transformer MoE (ICLR 2026)

ProMoE introduces a two‑step routing MoE framework with explicit semantic guidance that tackles the high spatial redundancy and functional heterogeneity of visual tokens, enabling diffusion transformers to scale efficiently and outperform dense models and prior MoE approaches across generation, convergence, and scaling benchmarks.

Explicit RoutingMixture of ExpertsPrototypical Routing
0 likes · 9 min read
ProMoE: Explicit Routing Breaks the Scaling Bottleneck of Diffusion‑Transformer MoE (ICLR 2026)
AIWalker
AIWalker
Mar 23, 2026 · Artificial Intelligence

Dynamic Dense Computing and Minimal End‑to‑End Design: YOLO-Master & YOLO26

By introducing a dynamic mixture‑of‑experts routing scheme and an end‑to‑end architecture that eliminates NMS and DFL, YOLO‑Master and YOLO26 dramatically cut compute waste and latency on edge devices, achieving up to 43% faster CPU inference while keeping model accuracy, with all code openly released.

Edge AIMixture of ExpertsYOLO
0 likes · 7 min read
Dynamic Dense Computing and Minimal End‑to‑End Design: YOLO-Master & YOLO26
AIWalker
AIWalker
Mar 7, 2026 · Artificial Intelligence

YOLO-Master v2026.02 Unveils Four Innovations for SOTA Object Detection

Tencent’s YOLO-Master v2026.02 adds a Mixture‑of‑Experts architecture, zero‑overhead LoRA fine‑tuning, Sparse SAHI inference for large images, and Cluster‑Weighted NMS, delivering 3‑5× faster inference, up to 70% reduced training resources, and markedly higher detection accuracy across diverse benchmarks.

LoRAMixture of ExpertsSparse Inference
0 likes · 15 min read
YOLO-Master v2026.02 Unveils Four Innovations for SOTA Object Detection
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Feb 23, 2026 · Artificial Intelligence

System Engineering Behind Billions of Parameters: Insider Training Details from Seven Top AI Labs

This article systematically dissects the engineering decisions behind frontier large‑language‑model training—covering architecture choices, attention variants, optimizer evolution, data‑curation strategies, scaling‑law insights, and post‑training SFT/RL pipelines—based on open‑source reports from seven leading AI laboratories.

Mixture of ExpertsOptimizerReinforcement Learning
0 likes · 26 min read
System Engineering Behind Billions of Parameters: Insider Training Details from Seven Top AI Labs
AI Engineering
AI Engineering
Feb 12, 2026 · Artificial Intelligence

MiniMax M2.5: 230B‑Parameter Model Activates 10B, Near Claude Sonnet for One‑Tenth the Cost

MiniMax’s new open‑source M2.5 model, built on a 230 billion‑parameter mixture‑of‑experts architecture that activates only 10 billion parameters per inference, delivers performance comparable to Claude Opus 4.6 across benchmarks, while costing roughly one‑tenth as much, and is already handling a large share of the company’s internal tasks.

AI AgentsClaude OpusMiniMax M2.5
0 likes · 6 min read
MiniMax M2.5: 230B‑Parameter Model Activates 10B, Near Claude Sonnet for One‑Tenth the Cost
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Feb 10, 2026 · Artificial Intelligence

Inside GLM-5: 745B Parameters, DeepSeek‑style Sparse Attention, and a 60% Stock Surge

The GLM-5 architecture, uncovered from a GitHub PR, doubles the previous model to 745 B parameters, adopts DeepSeek‑V3 sparse attention and multi‑token prediction, features a 78‑layer MoE with 256 experts, supports a 202K‑token context window, and its rumored test model "Pony Alpha" sparked a 60% rise in Zhipu AI's stock amid a crowded AI release season.

AI Stock ImpactDeepSeekGLM-5
0 likes · 6 min read
Inside GLM-5: 745B Parameters, DeepSeek‑style Sparse Attention, and a 60% Stock Surge
AI2ML AI to Machine Learning
AI2ML AI to Machine Learning
Feb 7, 2026 · Artificial Intelligence

Why the ‘Skills’ Approach Is the Third Major Compromise Shaping Enterprise AI in 2026

The article argues that embracing the Skills paradigm— a lightweight, low‑cost alternative to large‑scale model training—represents the third major compromise in the large‑model era, balancing reduced emergence and planning hallucinations against increased stability and engineering efficiency for enterprise AI deployments.

Agentic AIEnterprise AIMixture of Experts
0 likes · 8 min read
Why the ‘Skills’ Approach Is the Third Major Compromise Shaping Enterprise AI in 2026
PaperAgent
PaperAgent
Jan 22, 2026 · Artificial Intelligence

How STEM Replaces MoE Routing with Simple Table Lookup for Faster Transformers

The article presents STEM, a method that transforms dense and MoE transformer architectures by converting the expert routing step into a static table‑lookup operation, achieving higher parameter efficiency, lower communication overhead, and improved interpretability while maintaining or boosting downstream task performance.

Embedding LookupMixture of ExpertsTransformer
0 likes · 6 min read
How STEM Replaces MoE Routing with Simple Table Lookup for Faster Transformers
Programmer's Advance
Programmer's Advance
Jan 21, 2026 · Artificial Intelligence

Why GLM‑4.7‑Flash Delivers 70B‑Level Performance with Only 30B Parameters

GLM‑4.7‑Flash, released by Zhipu AI on Jan 20 2026, uses a Mixture‑of‑Experts (MoE) backbone and a Multi‑Latent Attention (MLA) mechanism to achieve near‑70B model quality with just 30 B total and 3 B active parameters, running on a single 24 GB GPU or even a Mac, while remaining fully open‑source and free to use.

AI model benchmarkGLM-4.7-FlashMixture of Experts
0 likes · 15 min read
Why GLM‑4.7‑Flash Delivers 70B‑Level Performance with Only 30B Parameters
AI Insight Log
AI Insight Log
Jan 20, 2026 · Artificial Intelligence

Is GLM-4.7-Flash the New 30B‑Level LLM King? Open‑Source and Ollama‑Ready

GLM‑4.7‑Flash, a 30B‑parameter MoE LLM released as fully open‑source and free, delivers 30B‑class performance across six benchmarks, runs locally with a single Ollama command, and offers a faster cloud‑hosted version with modest token‑based pricing, though hardware costs still apply.

Anthropic APIBenchmarkGLM-4.7-Flash
0 likes · 7 min read
Is GLM-4.7-Flash the New 30B‑Level LLM King? Open‑Source and Ollama‑Ready
JD Tech
JD Tech
Jan 13, 2026 · Artificial Intelligence

Mastering Large Language Models: Transformers, Scaling Laws, and MoE Explained

This extensive guide walks readers through the fundamentals of large language models, covering transformer architecture, pre‑training and fine‑tuning techniques, scaling laws, emergent abilities, mixture‑of‑experts designs, and practical comparisons, providing clear explanations, code snippets, and visual illustrations for deep learning practitioners.

Fine-tuningMixture of ExpertsPretraining
0 likes · 47 min read
Mastering Large Language Models: Transformers, Scaling Laws, and MoE Explained
Baobao Algorithm Notes
Baobao Algorithm Notes
Dec 25, 2025 · Artificial Intelligence

TeleChat3-105B: China’s First 100B‑Scale MoE Model and Its Technical Breakthroughs

The article analyzes TeleChat3-105B-A4.7-Thinking, the first domestically built 100‑billion‑parameter Mixture‑of‑Experts model, detailing its multi‑dimensional evaluation, three‑stage training pipeline, hardware‑level optimizations, fine‑grained architecture, and its significance for the evolving AI competition landscape.

Chinese AIMixture of ExpertsTeleChat3
0 likes · 6 min read
TeleChat3-105B: China’s First 100B‑Scale MoE Model and Its Technical Breakthroughs
HyperAI Super Neural
HyperAI Super Neural
Dec 19, 2025 · Artificial Intelligence

Weekly AI Paper Digest: Open-Source LLMs, Agent Systems, and Long-Context Reasoning

This week’s AI paper roundup reviews six recent research works—including RecGPT‑V2, Nemotron 3 Nano, FrontierScience benchmark, AutoGLM, Deeper‑GXX, and QwenLong‑L1.5—highlighting advances in large‑language‑model‑driven recommendation, Mixture‑of‑Experts models, expert‑level scientific reasoning, GUI‑based foundation agents, graph neural network deepening, and ultra‑long‑context inference.

AI researchAgent SystemsBenchmark
0 likes · 6 min read
Weekly AI Paper Digest: Open-Source LLMs, Agent Systems, and Long-Context Reasoning
AI Frontier Lectures
AI Frontier Lectures
Dec 9, 2025 · Artificial Intelligence

Can Token‑Level Surrogates Stabilize RL for Large Language Models? A Deep Dive

This article analyzes why optimizing sequence‑level rewards for LLMs with token‑level surrogate objectives can improve reinforcement‑learning stability, explains the theoretical conditions required, introduces Routing Replay for MoE models, and presents extensive experiments validating the approach.

Importance SamplingMixture of ExpertsReinforcement Learning
0 likes · 12 min read
Can Token‑Level Surrogates Stabilize RL for Large Language Models? A Deep Dive
PaperAgent
PaperAgent
Dec 4, 2025 · Artificial Intelligence

Mistral 3 Unveiled: How Its New Open‑Source Models Redefine Performance and Cost

Mistral AI’s latest open‑source release, Mistral 3, introduces three compact dense models and the powerful Mistral Large 3 MoE model, outperforming domestic rivals in benchmarks, offering strong multilingual and multimodal capabilities, and delivering the lowest cost‑performance ratio among open‑source LLMs.

Edge AIMistral 3Mixture of Experts
0 likes · 4 min read
Mistral 3 Unveiled: How Its New Open‑Source Models Redefine Performance and Cost
AI2ML AI to Machine Learning
AI2ML AI to Machine Learning
Dec 3, 2025 · Artificial Intelligence

2026 Forecast: How Large‑Model AI Will Evolve After 2025 Breakthroughs

The article reviews the major 2025 breakthroughs in multimodal, open‑source, and deployment technologies for large models and outlines four 2026 trends—including ToC vs. ToB service split, dual‑hand data generation, MoE routing advances, and AI4Science breakthroughs—that will shape the next wave of AI development.

AI DeploymentAI4ScienceMixture of Experts
0 likes · 6 min read
2026 Forecast: How Large‑Model AI Will Evolve After 2025 Breakthroughs
AntTech
AntTech
Nov 11, 2025 · Artificial Intelligence

Breaking the Efficiency Wall: Ant Group’s Bailing Model Paves the Way to AGI

At CNCC 2025, Ant Group’s Vice President Zhou Jun outlined the Bailing large‑model’s five‑layer architecture, hybrid linear attention, Ling Scaling Law, and novel training algorithms that dramatically cut costs and latency, achieving state‑of‑the‑art performance on math and code benchmarks while promoting open‑source collaboration toward AGI.

AGIMixture of Expertslarge language models
0 likes · 8 min read
Breaking the Efficiency Wall: Ant Group’s Bailing Model Paves the Way to AGI
Tencent Technical Engineering
Tencent Technical Engineering
Nov 10, 2025 · Artificial Intelligence

How Large Language Models Evolved in 2025: From DeepSeek to Kimi‑K2 and Beyond

This article maps the rapid evolution of open‑source large language models in 2025, explains the underlying architectural breakthroughs such as MLA, MoE, and NSA, compares dozens of models—including DeepSeek‑V3, OLMo2, Gemma3, Llama4, Qwen3, and Kimi‑K2—and highlights the emergence of powerful AI assistants like Dola, providing developers with a concise technical roadmap.

AI assistantLLM efficiencyMixture of Experts
0 likes · 44 min read
How Large Language Models Evolved in 2025: From DeepSeek to Kimi‑K2 and Beyond
DataFunTalk
DataFunTalk
Nov 10, 2025 · Artificial Intelligence

How Open-Source AI Models Are Outperforming Closed Giants on Cost and Performance

The article examines how open‑source models like DeepSeek‑R1 and Kimi K2 Thinking are challenging the traditional closed‑source, high‑capital AI paradigm by achieving comparable or superior benchmark results at a fraction of the training cost, reshaping market expectations, investment strategies, and the economics of AI development.

AI market dynamicsMixture of Expertsbenchmark performance
0 likes · 11 min read
How Open-Source AI Models Are Outperforming Closed Giants on Cost and Performance
Radish, Keep Going!
Radish, Keep Going!
Nov 4, 2025 · Artificial Intelligence

What You Need to Know: Backpropagation, FreeBSD, AI MoE, and More Tech Insights

This roundup covers essential insights on backpropagation fundamentals, FreeBSD self‑hosting benefits, an open‑source 30B MoE AI model, misuse of cybercrime laws, historic moving sidewalks, party‑planning hacks, deceptive signal‑strength tricks, a 1000‑hp micro motor, Nextcloud performance fixes, and Google Cloud account suspensions, offering a blend of technical depth and practical advice.

AIBackpropagationCloud Computing
0 likes · 11 min read
What You Need to Know: Backpropagation, FreeBSD, AI MoE, and More Tech Insights
Fighter's World
Fighter's World
Oct 25, 2025 · Artificial Intelligence

Rationally Understanding AI Capability Limits: Jason Wei’s Framework from Stanford

Jason Wei’s Stanford AI Club talk outlines three analytical ideas—Intelligence as a Commodity, Verifier's Law, and the Jagged Edge of Intelligence—to help businesses rationally assess AI’s economic shape, verification dynamics, and uneven performance across tasks.

Adaptive ComputationArtificial IntelligenceIntelligence as a Commodity
0 likes · 23 min read
Rationally Understanding AI Capability Limits: Jason Wei’s Framework from Stanford
Meituan Technology Team
Meituan Technology Team
Sep 11, 2025 · Artificial Intelligence

How LongCat-Flash Achieves Ultra-Fast, Low-Cost AI Agent Inference with SGLang

LongCat-Flash, an open‑source Mixture‑of‑Experts model released by Meituan, leverages model‑system co‑design, PD‑disaggregation, SBO scheduling and large‑scale expert parallelism within the SGLang framework to deliver dramatically lower latency, higher throughput and cost‑effective inference for AI agents, with detailed deployment instructions provided.

LongCat-FlashMixture of ExpertsSGLang
0 likes · 15 min read
How LongCat-Flash Achieves Ultra-Fast, Low-Cost AI Agent Inference with SGLang