BestHub
Sign in
Tagged articles

Mixture of Experts

168 articles · Page 1 of 2
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Aug 17, 2026 · Artificial Intelligence

N Load‑Balancing Techniques for Large‑Scale MoE Models

The article explains why token‑to‑expert routing in large Mixture‑of‑Experts models can become highly imbalanced, outlines the resulting compute bottlenecks, token overflow and expert degradation, and then surveys eight concrete load‑balancing strategies—ranging from importance‑based auxiliary losses to capacity‑aware token dropping and loss‑free bias control—detailing their mechanisms, trade‑offs, and practical considerations.

Auxiliary LossBalanced AssignmentCapacity Factor
0 likes · 17 min read
N Load‑Balancing Techniques for Large‑Scale MoE Models
TechVision Expert Circle
TechVision Expert Circle
Aug 17, 2026 · Artificial Intelligence

Open-Source LLMs Close the Gap: Low-Cost AI Poised to Redefine the Market

In early 2026, open-weight LLMs such as Llama 4, Qwen 3 and DeepSeek‑V3/R2 began matching or surpassing leading closed models like GPT‑4.5 and Claude Opus 5, driven by MoE architectures, FP8 precision, GRPO training and aggressive inference optimizations, prompting a reassessment of enterprise AI strategy.

AI economicsLarge Language ModelsMixture of Experts
0 likes · 15 min read
Open-Source LLMs Close the Gap: Low-Cost AI Poised to Redefine the Market
21CTO
21CTO
Aug 9, 2026 · Artificial Intelligence

ByteDance’s 10‑Trillion‑Parameter Gamble: Inside China’s Largest AI Model Pre‑training

ByteDance is reportedly pre‑training a 10‑trillion‑parameter AI model that dwarfs domestic rivals, uses a Mixture‑of‑Experts architecture with low activation ratios, demands roughly 36 000 Blackwell GPUs and $2.5 billion in hardware, and deliberately avoids distilling competitor models, raising questions about China’s AI frontier timeline.

AI scalingByteDanceDistillation avoidance
0 likes · 5 min read
ByteDance’s 10‑Trillion‑Parameter Gamble: Inside China’s Largest AI Model Pre‑training
21CTO
21CTO
Aug 8, 2026 · Artificial Intelligence

Jeff Dean Discusses the Next Decade of AI Just 12 Hours After Leaving Google

In a Stanford interview hosted by Dawn Song, Jeff Dean reflects on the origins of MoE, the transformative impact of deep learning, lessons from TensorFlow, how to spot breakthrough directions, AI agent risks, and his new venture Discovery Loop that aims to automate scientific research.

AI safetyArtificial IntelligenceDiscovery Loop
0 likes · 12 min read
Jeff Dean Discusses the Next Decade of AI Just 12 Hours After Leaving Google
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Aug 5, 2026 · Artificial Intelligence

Inside K3: How Stable Latent MoE and MLA Attention Are Designed

The article examines K3’s architecture—combining KDA, MLA, Stable Latent MoE and AttnRes—detailing the replacement of SwiGLU with SiTU‑GLU, the addition of RMSNorm for training stability, the Quantile Balancing load‑balancing scheme, and the trade‑offs behind its MLA and NoPE attention choices.

K3MLA AttentionMixture of Experts
0 likes · 15 min read
Inside K3: How Stable Latent MoE and MLA Attention Are Designed
21CTO
21CTO
Aug 3, 2026 · Artificial Intelligence

JetBrains’ Open‑Source 12B Code Model Mellum2: Private Deployment and High‑Throughput Over Claude Code

JetBrains released the open‑source 12‑billion‑parameter Mellum2 model, a MoE‑based code AI that delivers private on‑prem deployment, high‑throughput inference, and strong code‑generation benchmarks, positioning it as a fast, specialized alternative to Claude Code and other proprietary models.

Mellum2Mixture of ExpertsSoftware Engineering
0 likes · 7 min read
JetBrains’ Open‑Source 12B Code Model Mellum2: Private Deployment and High‑Throughput Over Claude Code
Black & White Path
Black & White Path
Aug 2, 2026 · Artificial Intelligence

Running a 2.8‑Trillion‑Parameter K3 Model on 4 GB VRAM with AirLLM

AirLLM introduces layer‑wise inference and per‑expert streaming to decouple VRAM usage from model size, enabling the 2.8‑trillion‑parameter Kimi K3 LLM to run on a single consumer‑grade GPU while preserving full‑precision accuracy and offering security‑focused insights.

AirLLMKimi K3Large Language Models
0 likes · 9 min read
Running a 2.8‑Trillion‑Parameter K3 Model on 4 GB VRAM with AirLLM
ITPUB
ITPUB
Jul 28, 2026 · Artificial Intelligence

Why Kimi K3’s Open‑Source Release Puts China at the Forefront of Global AI

Kimi K3, a 2.8‑trillion‑parameter MoE model with a 100 k‑token context, has been fully open‑sourced along with its weights, technical report and infra (MoonEP, FlashKDA, AgentEnv), delivering programming and agent benchmark results that rival top closed models such as Claude Fable 5 and GPT‑5.6 while sparking debate over alleged distillation and emphasizing AI safety and open‑weight governance.

AI benchmarksAI safetyKimi K3
0 likes · 11 min read
Why Kimi K3’s Open‑Source Release Puts China at the Forefront of Global AI
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Jul 27, 2026 · Artificial Intelligence

Scaling Residual Streams Efficiently: From DeepSeek mHC to xHC’s 16‑Stream Expansion

The blog details how xHC expands language‑model residual streams to 16, achieving nearly double the gain of DeepSeek mHC on 18B and 28B MoE models, and explains the design of Temporal Feature Augmentation and Sparse Write that make large‑N scaling both effective and affordable.

Large Language ModelsMixture of ExpertsScaling
0 likes · 24 min read
Scaling Residual Streams Efficiently: From DeepSeek mHC to xHC’s 16‑Stream Expansion
AI Programming Lab
AI Programming Lab
Jul 21, 2026 · Artificial Intelligence

How Kimi K3 and Qwen3.8‑Max Reach 2T+ Parameters: The Evolution of Chinese LLM Architecture

The article examines how Chinese LLMs such as Kimi K3 (2.8 T) and Qwen 3.8‑Max (2.4 T) achieved 2‑trillion‑parameter scales by progressively decoupling parameters from compute with Mixture‑of‑Experts, optimizing attention, and introducing training tricks like Muon and low‑precision quantization, tracing four years of architectural advances.

Attention MechanismsLarge Language ModelsMixture of Experts
0 likes · 11 min read
How Kimi K3 and Qwen3.8‑Max Reach 2T+ Parameters: The Evolution of Chinese LLM Architecture
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Jul 19, 2026 · Artificial Intelligence

How Kimi K3 Highlights Latent MoE as the Next Turning Point in Mixture‑of‑Experts Architecture

Latent MoE, demonstrated by NVIDIA’s Nemotron 3 Super and Moonshot AI’s 2.8 T‑parameter Kimi K3, compresses expert computations into a lower‑dimensional latent space, cutting memory reads and All‑to‑All traffic by fourfold, enabling more experts per token, higher accuracy, and up to 3.5× faster inference.

AI model scalingKimi K3Latent MoE
0 likes · 10 min read
How Kimi K3 Highlights Latent MoE as the Next Turning Point in Mixture‑of‑Experts Architecture
21CTO
21CTO
Jul 17, 2026 · Artificial Intelligence

Kimi K3 Unveiled: 2.8 Trillion‑Parameter Open‑Source LLM Sets New Record

On July 16, the Moon‑of‑Darkness team released Kimi K3, a 2.8‑trillion‑parameter open‑source large language model that introduces mixed‑linear attention, attention residuals, and a highly efficient Mixture‑of‑Experts design, achieving roughly 2.5× the scaling efficiency of its predecessor while approaching the performance of top closed‑source models.

Kimi K3Mixture of ExpertsOpen source AI
0 likes · 6 min read
Kimi K3 Unveiled: 2.8 Trillion‑Parameter Open‑Source LLM Sets New Record
SuanNi
SuanNi
Jul 17, 2026 · Artificial Intelligence

Kimi K3: The World’s First 3‑Trillion‑Parameter Open‑Source Model

Kimi K3, a 2.8‑trillion‑parameter open‑source LLM, outperforms top closed‑source models in benchmarks, excels at long‑range coding, GPU kernel optimization, and multimodal tasks, while introducing novel attention mechanisms, a compact Triton‑like compiler, and even a prototype ASIC chip.

GPU compilationKimi K3Mixture of Experts
0 likes · 9 min read
Kimi K3: The World’s First 3‑Trillion‑Parameter Open‑Source Model
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Jul 16, 2026 · Artificial Intelligence

Inkling – A 975‑Billion‑Parameter Open‑Weight Multimodal Model from Thinking Machines

Thinking Machines Lab unveiled Inkling, a 975‑billion‑parameter open‑weight multimodal model featuring a hybrid‑expert Transformer, 1‑million‑token context, and extensive benchmark results, alongside the lighter Inkling‑Small, with detailed architecture, training methodology, reinforcement‑learning enhancements, and practical examples of web‑app generation and tool‑calling.

InklingMixture of ExpertsMultimodal
0 likes · 15 min read
Inkling – A 975‑Billion‑Parameter Open‑Weight Multimodal Model from Thinking Machines
TechVision Expert Circle
TechVision Expert Circle
Jul 16, 2026 · Artificial Intelligence

Enterprise AI Trends for H2 2026: Key Priorities for Tech Leaders

In the second half of 2026, enterprise AI shifts from adoption to reliable, cost‑effective deployment, with six key trends—including multi‑agent orchestration, GraphRAG retrieval, MoE model clusters, AI observability, built‑in data governance, and reorganized AI engineering roles—guiding tech leaders toward trustworthy AI systems.

AI AgentAI ObservabilityAI Team Structure
0 likes · 13 min read
Enterprise AI Trends for H2 2026: Key Priorities for Tech Leaders
Architecture Digest
Architecture Digest
Jul 14, 2026 · Artificial Intelligence

How DeepSeek V4’s CSA+HCA Architecture Turns Million‑Token Context into a Cheap Feature

DeepSeek V4 introduces a hybrid CSA/HCA attention mechanism and a Mixture‑of‑Experts design that reduces inference cost to 27% of its predecessor, enables a 1‑million‑token context as a standard feature, and runs efficiently on both NVIDIA GPUs and Huawei Ascend NPU, dramatically lowering the barrier for Java developers to integrate large‑scale AI.

CSADeepSeek V4HCA
0 likes · 10 min read
How DeepSeek V4’s CSA+HCA Architecture Turns Million‑Token Context into a Cheap Feature
Black & White Path
Black & White Path
Jul 14, 2026 · Artificial Intelligence

SuperGemma 26B: The Fully Uncensored ‘Zero‑Guardrails’ AI Model Explained

Independent developer David Ondrej released SuperGemma 26B, an uncensored fork of Google’s Gemma 4 26B that removes all safety guardrails, runs locally on consumer‑grade GPUs, and has sparked intense debate over its technical merits, deployment simplicity, and the security risks of a truly unrestricted AI model.

AI safetyGemma 4Local Inference
0 likes · 11 min read
SuperGemma 26B: The Fully Uncensored ‘Zero‑Guardrails’ AI Model Explained
Geek Labs
Geek Labs
Jul 14, 2026 · Artificial Intelligence

How Colibri Runs a 744 B MoE Model on CPU with Only 25 GB RAM

Colibri demonstrates that the 744‑billion‑parameter GLM‑5.2 MoE model can be executed on a pure‑CPU system using roughly 25 GB of memory by streaming expert weights from disk, int4‑quantizing dense parameters, and employing a compact C inference engine.

CPU inferenceGLM-5.2MLA Attention
0 likes · 7 min read
How Colibri Runs a 744 B MoE Model on CPU with Only 25 GB RAM
Machine Heart
Machine Heart
Jul 13, 2026 · Artificial Intelligence

Breaking the Forgetting Barrier: CaRE Scales Continual Learning to 300+ Tasks

The paper introduces CaRE, a scalable continual‑learning framework that leverages a bi‑level routing mixture‑of‑experts to successfully train Vision Transformers on over 300 non‑overlapping tasks, outperforming existing baselines and accompanied by a new 1,000‑class benchmark, OmniBenchmark‑1K.

Bi-Level RoutingCaREMixture of Experts
0 likes · 11 min read
Breaking the Forgetting Barrier: CaRE Scales Continual Learning to 300+ Tasks
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Jul 10, 2026 · Artificial Intelligence

World’s First Embodied‑Native Action Model: Inside LingBot‑VA 2.0

LingBot‑VA 2.0 introduces the industry’s first embodied‑native pre‑trained robot brain, combining causal action modeling, a sparse MoE backbone, a semantic VAE tokenizer and asynchronous foresight reasoning to achieve six‑fold inference speedup, single‑GPU deployment and a 93.6% success rate on the RoboTwin 2.0 benchmark.

Causal ModelingForesight ReasoningMixture of Experts
0 likes · 11 min read
World’s First Embodied‑Native Action Model: Inside LingBot‑VA 2.0
Machine Heart
Machine Heart
Jul 9, 2026 · Artificial Intelligence

LingBot-Video: The Open‑Source Video Foundation Model Tailored for Robots

LingBot-Video is an open‑source video‑generation foundation model built for embodied AI, combining a sparse‑Mixture‑of‑Experts architecture, a multi‑stage data curriculum and six‑dimensional reward learning to achieve physically consistent video synthesis that outperforms existing open‑source baselines in both visual quality and robotic relevance.

Mixture of ExpertsMultimodal ModelVideo Generation
0 likes · 22 min read
LingBot-Video: The Open‑Source Video Foundation Model Tailored for Robots
Machine Heart
Machine Heart
Jul 8, 2026 · Artificial Intelligence

How LingBot‑VLA 2.0 Powers 20 Robot Configurations with an Open‑Source Embodied Brain

LingBot‑VLA 2.0 introduces a token‑level loss‑free MoE, dual‑query distillation, and a 60k‑hour heterogeneous dataset to achieve cross‑embodiment visual‑language‑action capabilities across 20 robot morphologies, delivering superior benchmark performance and sub‑130 ms inference while being fully open‑sourced.

Data EngineeringFuture PredictionMixture of Experts
0 likes · 16 min read
How LingBot‑VLA 2.0 Powers 20 Robot Configurations with an Open‑Source Embodied Brain
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Jul 1, 2026 · Artificial Intelligence

SAME: Stabilizing MoE to Reduce Dual Forgetting in Multimodal Continual Instruction Tuning

The paper identifies routing drift and expert drift as the two main causes of forgetting in multimodal continual instruction tuning (MCIT) and proposes SAME, which combines spectral‑aware routing, curvature‑aware scaling, and adaptive expert activation to keep MoE models stable, efficient, and less forgetful across long task sequences.

ICML 2026Instruction TuningMixture of Experts
0 likes · 19 min read
SAME: Stabilizing MoE to Reduce Dual Forgetting in Multimodal Continual Instruction Tuning
DeepHub IMBA
DeepHub IMBA
Jun 22, 2026 · Artificial Intelligence

How DiffusionGemma Shifts LLM Inference Bottleneck from Memory Bandwidth to Compute

DiffusionGemma, an experimental discrete text diffusion model built on the 26B MoE Gemma‑4 architecture, generates whole 256‑token blocks with bidirectional attention, moving the inference bottleneck from memory bandwidth to GPU compute, achieving up to four‑fold speed gains on H100 and RTX 5090 GPUs, though with lower output quality than standard autoregressive models.

DiffusionGemmaGPU performanceLLM inference
0 likes · 7 min read
How DiffusionGemma Shifts LLM Inference Bottleneck from Memory Bandwidth to Compute
AI Engineer Programming
AI Engineer Programming
Jun 17, 2026 · Artificial Intelligence

Local LLMs Viable: Sparse Attention, MoE, KV Compression, Multi‑Token Prediction

In early 2026, open‑source local large language models become practical alternatives thanks to sparse attention, MoE routing, latent KV compression, multi‑token prediction, and 4‑bit quantization, while hardware memory shortages and benchmark gaps with closed‑source models shape their deployment choices.

4-bit quantizationKV compressionLocal LLM
0 likes · 13 min read
Local LLMs Viable: Sparse Attention, MoE, KV Compression, Multi‑Token Prediction
AI Insight Log
AI Insight Log
Jun 12, 2026 · Artificial Intelligence

Kimi K2.7 Code: 1T MoE Model Cuts Tokens 30% and Beats Claude Opus on MCP Calls

The newly released Kimi K2.7 Code, a 1‑trillion‑parameter mixture‑of‑experts model that activates only 32 B parameters per inference, offers a 256 K context window, supports multimodal input, improves benchmark scores by up to 31.5 % over K2.6, reduces inference token usage by about 30 %, and achieves an 81.1 MCP tool‑call score surpassing Claude Opus 4.8, while providing a CLI installation command and usage guidelines.

Coding ModelKimiMCP
0 likes · 7 min read
Kimi K2.7 Code: 1T MoE Model Cuts Tokens 30% and Beats Claude Opus on MCP Calls
HyperAI Super Neural
HyperAI Super Neural
Jun 12, 2026 · Artificial Intelligence

DiffusionGemma Boosts Text Generation Speed Up to 4× with Discrete Diffusion

Google’s open‑source DiffusionGemma model leverages a 26‑billion‑parameter Mixture‑of‑Experts architecture and discrete diffusion decoding to generate whole text blocks, achieving up to four times faster generation—over 1100 tokens/s on an NVIDIA H100 and 700 tokens/s on an RTX 5090—while activating only 3.8 billion parameters during inference.

DiffusionGemmaDiscrete DiffusionGPU Acceleration
0 likes · 4 min read
DiffusionGemma Boosts Text Generation Speed Up to 4× with Discrete Diffusion
Machine Heart
Machine Heart
Jun 11, 2026 · Artificial Intelligence

Google Releases DiffusionGemma 26B MoE—Text Generation Up to 4× Faster

DiffusionGemma, Google's new 26‑billion‑parameter Mixture‑of‑Experts model, replaces token‑by‑token autoregression with a diffusion‑style output head that generates whole text blocks, delivering up to four‑fold speed gains on consumer GPUs while offering bidirectional attention and self‑correction, albeit with lower quality than standard Gemma 4.

DiffusionGemmaGPU AccelerationLocal Inference
0 likes · 6 min read
Google Releases DiffusionGemma 26B MoE—Text Generation Up to 4× Faster
Data Party THU
Data Party THU
Jun 6, 2026 · Artificial Intelligence

How a 400B MoE Model Runs on iPhone 17 Pro with Flash‑MoE

The article details how the open‑source Flash‑MoE engine enables the 400B‑parameter Qwen3.5‑397B‑A17B mixture‑of‑experts model to run on an iPhone 17 Pro, achieving about 0.6 tokens per second through a custom Metal pipeline, GCD‑driven SSD streaming, and aggressive caching strategies.

400BFlash-MoELLM inference
0 likes · 6 min read
How a 400B MoE Model Runs on iPhone 17 Pro with Flash‑MoE
Data Party THU
Data Party THU
May 29, 2026 · Artificial Intelligence

Token Superposition Training: 2.5× Faster LLM Pre‑training Without Model Changes

The article presents Token Superposition Training (TST), which temporarily averages embeddings of non‑overlapping token bags and predicts groups of tokens in a first phase before reverting to standard token‑wise prediction, achieving up to 2.5× pre‑training speedup on 10B‑1B MoE models without altering model architecture or inference.

LLM pretrainingMCE lossMixture of Experts
0 likes · 9 min read
Token Superposition Training: 2.5× Faster LLM Pre‑training Without Model Changes
Architect's Guide
Architect's Guide
May 29, 2026 · Artificial Intelligence

What Makes DeepSeek V4 Different? A Deep Technical Dive into Its Innovations

DeepSeek V4 introduces a suite of architectural breakthroughs—including mixed‑expert MoE, manifold‑constrained hyper‑connections, CSA/HCA hybrid attention, and FP4 quantization—that slash inference cost by up to tenfold while delivering million‑token context, competitive benchmarks, dual model variants, and a disruptive pricing strategy.

AI model benchmarkDeepSeek V4FP4 quantization
0 likes · 41 min read
What Makes DeepSeek V4 Different? A Deep Technical Dive into Its Innovations
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
May 22, 2026 · Artificial Intelligence

20‑Year‑Old Transformer Co‑author Open‑Sources a 218‑Billion‑Parameter Model

Cohere’s Command A+ model, built by Transformer co‑author Aidan Gomez and backed by Nick Frosst, packs 218 billion parameters but activates only 25 billion at inference, uses a lossless 4‑bit quantization scheme, offers native citation support, runs on a single B200 or two H100 GPUs, and is released under an Apache 2.0 license, marking a major shift toward truly open‑source, enterprise‑ready large language models.

AIApache 2.0Cohere
0 likes · 12 min read
20‑Year‑Old Transformer Co‑author Open‑Sources a 218‑Billion‑Parameter Model
Machine Heart
Machine Heart
May 22, 2026 · Artificial Intelligence

Breaking the Echo Chamber: MP‑MoE Introduces Ensemble‑Pruning for Diverse Experts

The paper presents MP‑MoE, a new Mixture‑of‑Experts architecture that replaces top‑k routing with Mahalanobis‑based ensemble pruning, explicitly encouraging expert diversity via a co‑occurrence matrix, and uses an efficient greedy algorithm with incremental Cholesky updates, achieving higher performance with minimal training overhead and no inference cost.

Dynamic RoutingEnsemble PruningExpert Diversity
0 likes · 8 min read
Breaking the Echo Chamber: MP‑MoE Introduces Ensemble‑Pruning for Diverse Experts
Alibaba International Intelligent Technology
Alibaba International Intelligent Technology
May 22, 2026 · Artificial Intelligence

How SORT Transforms Precision Ranking with a Transformer‑Based Architecture

SORT re‑architects industrial‑scale ranking by shifting to a request‑centric data paradigm, integrating sparse and MoE optimizations into a Transformer backbone, and delivering significant CTR‑AUC, FLOPs, and online metric improvements while maintaining high training and inference efficiency.

Large Model ScalingMixture of ExpertsRanking
0 likes · 19 min read
How SORT Transforms Precision Ranking with a Transformer‑Based Architecture
Machine Heart
Machine Heart
May 20, 2026 · Artificial Intelligence

Can Tabular Anomaly Detection Move Beyond One‑for‑One? OFA‑TAD Introduces a One‑for‑All Paradigm

Tabular anomaly detection traditionally requires training a separate model for each dataset (one‑for‑one), but the new OFA‑TAD framework trains once on multiple source tables and directly transfers to unseen target tables without fine‑tuning, leveraging multi‑view distance encoding, MoE fusion, and synthetic pseudo‑anomalies to achieve state‑of‑the‑art performance across 34 datasets in 14 domains.

Mixture of ExpertsOFA-TADmulti-view distance
0 likes · 10 min read
Can Tabular Anomaly Detection Move Beyond One‑for‑One? OFA‑TAD Introduces a One‑for‑All Paradigm
Data Party THU
Data Party THU
May 17, 2026 · Artificial Intelligence

How DeepSeek Leverages MoE Parallelism: GPU Compute and Communication Optimizations

The article dissects DeepSeek's MoE model‑parallel strategy, explaining how GPU compute and communication are overlapped through expert, pipeline, and ZeRO‑1 parallelism, and introduces DualPipe and Waved‑EP kernels that enable efficient training on large‑scale hardware.

DeepSeekGPU Communication OverlapMixture of Experts
0 likes · 18 min read
How DeepSeek Leverages MoE Parallelism: GPU Compute and Communication Optimizations
PaperAgent
PaperAgent
May 13, 2026 · Artificial Intelligence

One-for-All Multi-Agent Collaboration: Adaptive Cross-Task Topology Design

The paper introduces OFA-MAS, a one‑for‑all multi‑agent system that learns a universal topology designer using task‑aware graph encoding and a Mixture‑of‑Experts generator, achieving superior performance, OOD generalization, robustness, and efficiency across six major benchmarks.

LLMMixture of ExpertsTask-Aware Graph Encoder
0 likes · 14 min read
One-for-All Multi-Agent Collaboration: Adaptive Cross-Task Topology Design
Lao Guo's Learning Space
Lao Guo's Learning Space
May 12, 2026 · Artificial Intelligence

Demystifying the Core Technologies Behind ChatGPT, GPT‑4, and DeepSeek

This article breaks down the key algorithms that power large‑language models—Transformer, Mixture‑of‑Experts, Flash Attention, KV‑Cache, Multi‑Token Prediction, quantization, Chain‑of‑Thought and Retrieval‑Augmented Generation—explaining how each contributes to the performance of ChatGPT, GPT‑4 and DeepSeek.

Flash AttentionKV cacheLarge Language Models
0 likes · 10 min read
Demystifying the Core Technologies Behind ChatGPT, GPT‑4, and DeepSeek
Old Zhang's AI Learning
Old Zhang's AI Learning
May 11, 2026 · Artificial Intelligence

Open‑Source Qwen3.6‑35B‑A3B Runs at 162 tok/s on a Single RTX 5090

The article introduces the open‑source Qwen3.6‑35B‑A3B model, explains its MoE architecture, three‑stage LoRA fine‑tuning, shows benchmark results where it achieves 161.9 tok/s on an RTX 5090—2.6× faster than a dense 27B counterpart—and discusses deployment tips, quantized GGUF release, and known compatibility pitfalls.

GGUF quantizationLoRA fine-tuningMixture of Experts
0 likes · 7 min read
Open‑Source Qwen3.6‑35B‑A3B Runs at 162 tok/s on a Single RTX 5090
Old Zhang's AI Learning
Old Zhang's AI Learning
May 7, 2026 · Artificial Intelligence

How Unsloth and NVIDIA Boost Consumer‑GPU LLM Training by ~25% with Three Simple Optimizations

Unsloth and NVIDIA identified three low‑level bottlenecks in LLM fine‑tuning on consumer GPUs—repeated packed‑sequence metadata construction, serialized copy‑and‑compute during gradient checkpointing, and per‑expert routing overhead in MoE—and applied targeted patches that together deliver roughly a 25% speedup without changing hardware, code, or frameworks.

GPU OptimizationGradient CheckpointingLLM training
0 likes · 12 min read
How Unsloth and NVIDIA Boost Consumer‑GPU LLM Training by ~25% with Three Simple Optimizations
AI Engineer Programming
AI Engineer Programming
May 7, 2026 · Artificial Intelligence

How Cursor Turned Its Coding Agent from Demo to Production

The article examines Cursor's journey of shipping its Composer coding agent, detailing the agentic AI model, system architecture, and the three major production challenges—diff handling, latency accumulation, and sandbox scaling—along with the engineering solutions that enabled reliable, fast, and adoptable AI‑driven code generation.

Coding AgentCursorMixture of Experts
0 likes · 16 min read
How Cursor Turned Its Coding Agent from Demo to Production
Machine Heart
Machine Heart
May 4, 2026 · Artificial Intelligence

Mega MoE vs SonicMoE: Which Will Lead the Next AI Speed Race?

SonicMoE, a new ultra‑fast Mixture‑of‑Experts model from Tri Dao and Ion Stoica’s team, achieves peak throughput on Nvidia Blackwell GPUs, outperforms DeepSeek’s DeepGEMM, and introduces algorithmic redesigns that decouple activation memory from expert granularity while fusing I/O‑aware kernels for up to double the speed of existing MoE frameworks.

AI performanceBlackwellGPU Acceleration
0 likes · 12 min read
Mega MoE vs SonicMoE: Which Will Lead the Next AI Speed Race?
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
May 3, 2026 · Artificial Intelligence

Running a 400B Mixture‑of‑Experts LLM on iPhone 17 Pro: Inside Flash‑MoE

The article details how the open‑source Flash‑MoE engine streams a 400‑billion‑parameter Mixture‑of‑Experts language model on an iPhone 17 Pro, achieving interactive‑level token throughput by eliminating Python dependencies, crafting a custom Metal pipeline, and streaming weights directly from SSD.

Apple SiliconFlash-MoEGCD
0 likes · 7 min read
Running a 400B Mixture‑of‑Experts LLM on iPhone 17 Pro: Inside Flash‑MoE
Data Party THU
Data Party THU
May 2, 2026 · Artificial Intelligence

Training an 11.5 B‑parameter Universal Interatomic Potential in Hours on Exascale Supercomputers

A Chinese Academy of Sciences team introduced the MatRIS‑MoE model and the Janus training framework, enabling a 11.5 billion‑parameter universal machine‑learning interatomic potential to be trained on two exascale systems at 1.2 EFLOPS, compressing weeks‑long training into a few hours.

AI for ScienceExascale trainingML interatomic potentials
0 likes · 8 min read
Training an 11.5 B‑parameter Universal Interatomic Potential in Hours on Exascale Supercomputers
Machine Heart
Machine Heart
May 1, 2026 · Artificial Intelligence

How a 400B Mixture‑of‑Experts Model Runs on the iPhone 17 Pro

The article details the Flash‑MoE project that streams the 400 billion‑parameter Qwen3.5‑397B‑A17B mixture‑of‑experts model on an iPhone 17 Pro, achieving up to 0.6 tokens per second with a custom Metal‑GPU pipeline, zero‑Python code, and SSD‑backed weight streaming that keeps only 5.5 GB in RAM.

Flash-MoELLMMetal
0 likes · 7 min read
How a 400B Mixture‑of‑Experts Model Runs on the iPhone 17 Pro
Machine Heart
Machine Heart
Apr 30, 2026 · Artificial Intelligence

Beyond DeepSeek V4: A Trillion‑Parameter LLM Trained End‑to‑End on Domestic Chips

The article analyzes how both DeepSeek V4 and Meituan's LongCat‑2.0‑P preview, each with trillion‑scale parameters and 1 M‑token context, were trained and inferred entirely on Chinese‑made accelerators, detailing memory optimizations, deterministic operators, MoE redesigns, and massive multi‑card clusters that prove domestic compute can meet top‑tier AI workloads.

Deterministic OpsDomestic AI ChipLongCat
0 likes · 13 min read
Beyond DeepSeek V4: A Trillion‑Parameter LLM Trained End‑to‑End on Domestic Chips
Architect's Must-Have
Architect's Must-Have
Apr 28, 2026 · Artificial Intelligence

Why DeepSeek V4 Stands Apart: A Deep Dive into Its Architecture and Performance

DeepSeek V4 introduces a suite of architectural innovations—including mixed attention, manifold‑constrained hyper‑connections, the Muon optimizer, and FP4‑aware quantization—that together slash million‑token inference cost to a tenth of its predecessor while delivering benchmark results that rival top‑tier closed‑source models.

DeepSeek V4EfficiencyFP4 quantization
0 likes · 44 min read
Why DeepSeek V4 Stands Apart: A Deep Dive into Its Architecture and Performance
CodeTrend
CodeTrend
Apr 26, 2026 · Artificial Intelligence

DeepSeek V4 Architecture: High‑Efficiency Long‑Context Model Design

DeepSeek V4, released in April 2026, introduces two versions—Pro and Flash—with up to 1.6 trillion parameters and a million‑token context window, leveraging hybrid attention, compressed KV cache, and specialized training techniques to dramatically cut hardware dependence and inference cost.

DeepSeekFP4Hybrid Attention
0 likes · 5 min read
DeepSeek V4 Architecture: High‑Efficiency Long‑Context Model Design
AI2ML AI to Machine Learning
AI2ML AI to Machine Learning
Apr 25, 2026 · Artificial Intelligence

How DeepSeek V4 Advances Structured Optimization in the Large‑Model Era

The article analyses DeepSeek V4’s architectural innovations—including Compressed Sparse Attention, Heavily Compressed Attention, a cross‑layer MoE design, and an Agent‑RL framework with Generative Reward Models and multi‑teacher distillation—while comparing its long‑context capabilities and efficiency to rival LLMs such as GLM, Kimi, Claude, GPT and Gemini.

Agent Reinforcement LearningCompressed Sparse AttentionDeepSeek V4
0 likes · 7 min read
How DeepSeek V4 Advances Structured Optimization in the Large‑Model Era
Architect's Tech Stack
Architect's Tech Stack
Apr 25, 2026 · Artificial Intelligence

DeepSeek‑V4 Launch: 1.6 T Parameters, 1 M‑Token Context, Programming Skills Lead Open‑Source Rankings

DeepSeek released the V4 series—V4‑Pro (1.6 T total, 49 B active) and V4‑Flash (284 B total, 13 B active)—featuring three architectural upgrades, three inference modes, mixed‑precision FP4/FP8 weights, and benchmark results that place its programming ability at the top of open‑source models while supporting a million‑token context window.

AI architectureDeepSeekMixture of Experts
0 likes · 5 min read
DeepSeek‑V4 Launch: 1.6 T Parameters, 1 M‑Token Context, Programming Skills Lead Open‑Source Rankings
ArcThink
ArcThink
Apr 25, 2026 · Artificial Intelligence

DeepSeek V4’s Silent Launch: 1.6 T Parameters, Triple Innovation, and Redefined Accessibility

DeepSeek V4 quietly debuted with a 1.6‑trillion‑parameter MoE model, introducing CSA+HCA compressed attention, mHC manifold‑constrained hyperconnections, and the Muon optimizer, achieving 1M‑token context at a quarter of V3’s cost, top Codeforces and LiveCodeBench scores, a 1/7 Opus price, MIT open‑source licensing, and dual‑stack Ascend NPU/NVIDIA GPU support.

DeepSeek V4Manifold-constrained HyperconnectionMixture of Experts
0 likes · 17 min read
DeepSeek V4’s Silent Launch: 1.6 T Parameters, Triple Innovation, and Redefined Accessibility
AI Agent Super App
AI Agent Super App
Apr 24, 2026 · Artificial Intelligence

DeepSeek V4 Launches with 1.6 T Parameters and 1 Million‑Token Context

DeepSeek V4, released on April 24 2026, offers two SKUs—Pro with 1.6 T total parameters and Flash with 284 B—both supporting a 1‑million‑token context window, ultra‑low inference cost, pricing as low as ¥0.2 per million tokens, Huawei Ascend deployment, and seamless OpenAI/Anthropic API compatibility.

AI pricingAPI CompatibilityDeepSeek
0 likes · 7 min read
DeepSeek V4 Launches with 1.6 T Parameters and 1 Million‑Token Context
Alibaba International Intelligent Technology
Alibaba International Intelligent Technology
Apr 24, 2026 · Artificial Intelligence

Alibaba International AI Team Lands Multiple Papers at SIGIR, WWW, and WSDM 2026

Alibaba International Intelligent Technology showcases nine industrial‑grade recommendation and search papers accepted at SIGIR, WWW, and WSDM 2026, detailing sparse scaling, counterfactual multi‑task learning, generative recommendation, MoE routing, and multimodal semantic ID breakthroughs with extensive offline and online results.

Generative RecommendationLarge Language ModelsMixture of Experts
0 likes · 24 min read
Alibaba International AI Team Lands Multiple Papers at SIGIR, WWW, and WSDM 2026
SuanNi
SuanNi
Apr 21, 2026 · Artificial Intelligence

How Qwen3.6‑35B‑A3B Matches Dense Models with Only 30 B Active Parameters

The article analyzes Qwen3.6‑35B‑A3B’s MoE architecture, showing how its 30 B active parameters outperform larger dense models across programming, agent, and multimodal benchmarks, and examines the flagship Qwen3.6‑Max‑Preview’s substantial gains in world knowledge, instruction following, and third‑party rankings.

AI evaluationMixture of ExpertsQwen
0 likes · 5 min read
How Qwen3.6‑35B‑A3B Matches Dense Models with Only 30 B Active Parameters
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Apr 21, 2026 · Artificial Intelligence

How a 22‑Year‑Old Reversed‑Engineered Mythos into OpenMythos Using MoE and DeepSeek‑Inspired Attention

OpenMythos re‑creates the Claude Mythos architecture as a Recurrent‑Depth Transformer with MoE routing, achieving comparable performance to larger Transformers while using roughly half the parameters, and demonstrates systematic generalization and depth extrapolation through looped inference in latent space.

AI architectureLooped Language ModelsMixture of Experts
0 likes · 6 min read
How a 22‑Year‑Old Reversed‑Engineered Mythos into OpenMythos Using MoE and DeepSeek‑Inspired Attention
AI Large-Model Wave and Transformation Guide
AI Large-Model Wave and Transformation Guide
Apr 18, 2026 · Artificial Intelligence

Does Qwen3.6‑35B‑A3B Really Outclass All AI Coding Models? Inside the Benchmark Breakdown

Qwen3.6‑35B‑A3B, a mixture‑of‑experts model that activates only 3 B parameters, outperforms leading AI systems across SWE‑bench, Terminal‑Bench, NL2Repo and several agentic coding benchmarks, while also achieving top scores in GPQA, HMMT and RealWorldQA, prompting a reassessment of domestic LLM capabilities.

AI codingAgentic codingChinese AI
0 likes · 7 min read
Does Qwen3.6‑35B‑A3B Really Outclass All AI Coding Models? Inside the Benchmark Breakdown
Machine Heart
Machine Heart
Apr 17, 2026 · Artificial Intelligence

DeepSeek Introduces Mega MoE and FP4 Indexer – Inside the New GPU Fusion Kernel

DeepSeek's latest DeepGEMM update adds Mega MoE, a fused GPU kernel that collapses the entire Mixture‑of‑Experts pipeline and overlaps computation with NVLink communication, while also unveiling an FP4 indexer and FP8×FP4 precision experiments, signaling a push toward highly efficient large‑scale AI training.

DeepGEMMDeepSeekFP4 Indexer
0 likes · 5 min read
DeepSeek Introduces Mega MoE and FP4 Indexer – Inside the New GPU Fusion Kernel
Machine Heart
Machine Heart
Mar 31, 2026 · Artificial Intelligence

ProMoE: Explicit Routing Breaks the Scaling Bottleneck of Diffusion‑Transformer MoE (ICLR 2026)

ProMoE introduces a two‑step routing MoE framework with explicit semantic guidance that tackles the high spatial redundancy and functional heterogeneity of visual tokens, enabling diffusion transformers to scale efficiently and outperform dense models and prior MoE approaches across generation, convergence, and scaling benchmarks.

Explicit RoutingMixture of ExpertsPrototypical Routing
0 likes · 9 min read
ProMoE: Explicit Routing Breaks the Scaling Bottleneck of Diffusion‑Transformer MoE (ICLR 2026)
AIWalker
AIWalker
Mar 23, 2026 · Artificial Intelligence

Dynamic Dense Computing and Minimal End‑to‑End Design: YOLO-Master & YOLO26

By introducing a dynamic mixture‑of‑experts routing scheme and an end‑to‑end architecture that eliminates NMS and DFL, YOLO‑Master and YOLO26 dramatically cut compute waste and latency on edge devices, achieving up to 43% faster CPU inference while keeping model accuracy, with all code openly released.

Dynamic RoutingMixture of ExpertsModel Optimization
0 likes · 7 min read
Dynamic Dense Computing and Minimal End‑to‑End Design: YOLO-Master & YOLO26
AIWalker
AIWalker
Mar 7, 2026 · Artificial Intelligence

YOLO-Master v2026.02 Unveils Four Innovations for SOTA Object Detection

Tencent’s YOLO-Master v2026.02 adds a Mixture‑of‑Experts architecture, zero‑overhead LoRA fine‑tuning, Sparse SAHI inference for large images, and Cluster‑Weighted NMS, delivering 3‑5× faster inference, up to 70% reduced training resources, and markedly higher detection accuracy across diverse benchmarks.

LoRAMixture of ExpertsModel Optimization
0 likes · 15 min read
YOLO-Master v2026.02 Unveils Four Innovations for SOTA Object Detection
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Feb 23, 2026 · Artificial Intelligence

System Engineering Behind Billions of Parameters: Insider Training Details from Seven Top AI Labs

This article systematically dissects the engineering decisions behind frontier large‑language‑model training—covering architecture choices, attention variants, optimizer evolution, data‑curation strategies, scaling‑law insights, and post‑training SFT/RL pipelines—based on open‑source reports from seven leading AI laboratories.

Large Language ModelsMixture of ExpertsOptimizer
0 likes · 26 min read
System Engineering Behind Billions of Parameters: Insider Training Details from Seven Top AI Labs
AI Engineering
AI Engineering
Feb 12, 2026 · Artificial Intelligence

MiniMax M2.5: 230B‑Parameter Model Activates 10B, Near Claude Sonnet for One‑Tenth the Cost

MiniMax’s new open‑source M2.5 model, built on a 230 billion‑parameter mixture‑of‑experts architecture that activates only 10 billion parameters per inference, delivers performance comparable to Claude Opus 4.6 across benchmarks, while costing roughly one‑tenth as much, and is already handling a large share of the company’s internal tasks.

AI agentsClaude OpusMiniMax M2.5
0 likes · 6 min read
MiniMax M2.5: 230B‑Parameter Model Activates 10B, Near Claude Sonnet for One‑Tenth the Cost
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Feb 10, 2026 · Artificial Intelligence

Inside GLM-5: 745B Parameters, DeepSeek‑style Sparse Attention, and a 60% Stock Surge

The GLM-5 architecture, uncovered from a GitHub PR, doubles the previous model to 745 B parameters, adopts DeepSeek‑V3 sparse attention and multi‑token prediction, features a 78‑layer MoE with 256 experts, supports a 202K‑token context window, and its rumored test model "Pony Alpha" sparked a 60% rise in Zhipu AI's stock amid a crowded AI release season.

AI Stock ImpactDeepSeekGLM-5
0 likes · 6 min read
Inside GLM-5: 745B Parameters, DeepSeek‑style Sparse Attention, and a 60% Stock Surge
AI2ML AI to Machine Learning
AI2ML AI to Machine Learning
Feb 7, 2026 · Artificial Intelligence

Why the ‘Skills’ Approach Is the Third Major Compromise Shaping Enterprise AI in 2026

The article argues that embracing the Skills paradigm— a lightweight, low‑cost alternative to large‑scale model training—represents the third major compromise in the large‑model era, balancing reduced emergence and planning hallucinations against increased stability and engineering efficiency for enterprise AI deployments.

Enterprise AILarge Language ModelsMixture of Experts
0 likes · 8 min read
Why the ‘Skills’ Approach Is the Third Major Compromise Shaping Enterprise AI in 2026
PaperAgent
PaperAgent
Jan 22, 2026 · Artificial Intelligence

How STEM Replaces MoE Routing with Simple Table Lookup for Faster Transformers

The article presents STEM, a method that transforms dense and MoE transformer architectures by converting the expert routing step into a static table‑lookup operation, achieving higher parameter efficiency, lower communication overhead, and improved interpretability while maintaining or boosting downstream task performance.

Embedding LookupMixture of ExpertsTransformer
0 likes · 6 min read
How STEM Replaces MoE Routing with Simple Table Lookup for Faster Transformers
Programmer's Advance
Programmer's Advance
Jan 21, 2026 · Artificial Intelligence

Why GLM‑4.7‑Flash Delivers 70B‑Level Performance with Only 30B Parameters

GLM‑4.7‑Flash, released by Zhipu AI on Jan 20 2026, uses a Mixture‑of‑Experts (MoE) backbone and a Multi‑Latent Attention (MLA) mechanism to achieve near‑70B model quality with just 30 B total and 3 B active parameters, running on a single 24 GB GPU or even a Mac, while remaining fully open‑source and free to use.

AI model benchmarkGLM-4.7-FlashMixture of Experts
0 likes · 15 min read
Why GLM‑4.7‑Flash Delivers 70B‑Level Performance with Only 30B Parameters
AI Insight Log
AI Insight Log
Jan 20, 2026 · Artificial Intelligence

Is GLM-4.7-Flash the New 30B‑Level LLM King? Open‑Source and Ollama‑Ready

GLM‑4.7‑Flash, a 30B‑parameter MoE LLM released as fully open‑source and free, delivers 30B‑class performance across six benchmarks, runs locally with a single Ollama command, and offers a faster cloud‑hosted version with modest token‑based pricing, though hardware costs still apply.

Anthropic APIGLM-4.7-FlashMixture of Experts
0 likes · 7 min read
Is GLM-4.7-Flash the New 30B‑Level LLM King? Open‑Source and Ollama‑Ready
JD Tech
JD Tech
Jan 13, 2026 · Artificial Intelligence

Mastering Large Language Models: Transformers, Scaling Laws, and MoE Explained

This extensive guide walks readers through the fundamentals of large language models, covering transformer architecture, pre‑training and fine‑tuning techniques, scaling laws, emergent abilities, mixture‑of‑experts designs, and practical comparisons, providing clear explanations, code snippets, and visual illustrations for deep learning practitioners.

Fine-tuningMixture of Expertsemergent abilities
0 likes · 47 min read
Mastering Large Language Models: Transformers, Scaling Laws, and MoE Explained
Baobao Algorithm Notes
Baobao Algorithm Notes
Dec 25, 2025 · Artificial Intelligence

TeleChat3-105B: China’s First 100B‑Scale MoE Model and Its Technical Breakthroughs

The article analyzes TeleChat3-105B-A4.7-Thinking, the first domestically built 100‑billion‑parameter Mixture‑of‑Experts model, detailing its multi‑dimensional evaluation, three‑stage training pipeline, hardware‑level optimizations, fine‑grained architecture, and its significance for the evolving AI competition landscape.

AI trainingChinese AIMixture of Experts
0 likes · 6 min read
TeleChat3-105B: China’s First 100B‑Scale MoE Model and Its Technical Breakthroughs
HyperAI Super Neural
HyperAI Super Neural
Dec 19, 2025 · Artificial Intelligence

Weekly AI Paper Digest: Open-Source LLMs, Agent Systems, and Long-Context Reasoning

This week’s AI paper roundup reviews six recent research works—including RecGPT‑V2, Nemotron 3 Nano, FrontierScience benchmark, AutoGLM, Deeper‑GXX, and QwenLong‑L1.5—highlighting advances in large‑language‑model‑driven recommendation, Mixture‑of‑Experts models, expert‑level scientific reasoning, GUI‑based foundation agents, graph neural network deepening, and ultra‑long‑context inference.

AI researchAgent SystemsLarge Language Models
0 likes · 6 min read
Weekly AI Paper Digest: Open-Source LLMs, Agent Systems, and Long-Context Reasoning
AI Frontier Lectures
AI Frontier Lectures
Dec 9, 2025 · Artificial Intelligence

Can Token‑Level Surrogates Stabilize RL for Large Language Models? A Deep Dive

This article analyzes why optimizing sequence‑level rewards for LLMs with token‑level surrogate objectives can improve reinforcement‑learning stability, explains the theoretical conditions required, introduces Routing Replay for MoE models, and presents extensive experiments validating the approach.

Importance SamplingLarge Language ModelsMixture of Experts
0 likes · 12 min read
Can Token‑Level Surrogates Stabilize RL for Large Language Models? A Deep Dive
PaperAgent
PaperAgent
Dec 4, 2025 · Artificial Intelligence

Mistral 3 Unveiled: How Its New Open‑Source Models Redefine Performance and Cost

Mistral AI’s latest open‑source release, Mistral 3, introduces three compact dense models and the powerful Mistral Large 3 MoE model, outperforming domestic rivals in benchmarks, offering strong multilingual and multimodal capabilities, and delivering the lowest cost‑performance ratio among open‑source LLMs.

Mistral 3Mixture of ExpertsModel Benchmark
0 likes · 4 min read
Mistral 3 Unveiled: How Its New Open‑Source Models Redefine Performance and Cost
AI2ML AI to Machine Learning
AI2ML AI to Machine Learning
Dec 3, 2025 · Artificial Intelligence

2026 Forecast: How Large‑Model AI Will Evolve After 2025 Breakthroughs

The article reviews the major 2025 breakthroughs in multimodal, open‑source, and deployment technologies for large models and outlines four 2026 trends—including ToC vs. ToB service split, dual‑hand data generation, MoE routing advances, and AI4Science breakthroughs—that will shape the next wave of AI development.

AI deploymentAI4ScienceLarge Language Models
0 likes · 6 min read
2026 Forecast: How Large‑Model AI Will Evolve After 2025 Breakthroughs
AntTech
AntTech
Nov 11, 2025 · Artificial Intelligence

Breaking the Efficiency Wall: Ant Group’s Bailing Model Paves the Way to AGI

At CNCC 2025, Ant Group’s Vice President Zhou Jun outlined the Bailing large‑model’s five‑layer architecture, hybrid linear attention, Ling Scaling Law, and novel training algorithms that dramatically cut costs and latency, achieving state‑of‑the‑art performance on math and code benchmarks while promoting open‑source collaboration toward AGI.

AGILarge Language ModelsMixture of Experts
0 likes · 8 min read
Breaking the Efficiency Wall: Ant Group’s Bailing Model Paves the Way to AGI
Tencent Technical Engineering
Tencent Technical Engineering
Nov 10, 2025 · Artificial Intelligence

How Large Language Models Evolved in 2025: From DeepSeek to Kimi‑K2 and Beyond

This article maps the rapid evolution of open‑source large language models in 2025, explains the underlying architectural breakthroughs such as MLA, MoE, and NSA, compares dozens of models—including DeepSeek‑V3, OLMo2, Gemma3, Llama4, Qwen3, and Kimi‑K2—and highlights the emergence of powerful AI assistants like Dola, providing developers with a concise technical roadmap.

AI assistantLLM efficiencyLarge Language Models
0 likes · 44 min read
How Large Language Models Evolved in 2025: From DeepSeek to Kimi‑K2 and Beyond
DataFunTalk
DataFunTalk
Nov 10, 2025 · Artificial Intelligence

How Open-Source AI Models Are Outperforming Closed Giants on Cost and Performance

The article examines how open‑source models like DeepSeek‑R1 and Kimi K2 Thinking are challenging the traditional closed‑source, high‑capital AI paradigm by achieving comparable or superior benchmark results at a fraction of the training cost, reshaping market expectations, investment strategies, and the economics of AI development.

AI market dynamicsMixture of ExpertsOpen source AI
0 likes · 11 min read
How Open-Source AI Models Are Outperforming Closed Giants on Cost and Performance
Radish, Keep Going!
Radish, Keep Going!
Nov 4, 2025 · Artificial Intelligence

What You Need to Know: Backpropagation, FreeBSD, AI MoE, and More Tech Insights

This roundup covers essential insights on backpropagation fundamentals, FreeBSD self‑hosting benefits, an open‑source 30B MoE AI model, misuse of cybercrime laws, historic moving sidewalks, party‑planning hacks, deceptive signal‑strength tricks, a 1000‑hp micro motor, Nextcloud performance fixes, and Google Cloud account suspensions, offering a blend of technical depth and practical advice.

AIBackpropagationCloud Computing
0 likes · 11 min read
What You Need to Know: Backpropagation, FreeBSD, AI MoE, and More Tech Insights
Fighter's World
Fighter's World
Oct 25, 2025 · Artificial Intelligence

Rationally Understanding AI Capability Limits: Jason Wei’s Framework from Stanford

Jason Wei’s Stanford AI Club talk outlines three analytical ideas—Intelligence as a Commodity, Verifier's Law, and the Jagged Edge of Intelligence—to help businesses rationally assess AI’s economic shape, verification dynamics, and uneven performance across tasks.

Adaptive ComputationArtificial IntelligenceIntelligence as a Commodity
0 likes · 23 min read
Rationally Understanding AI Capability Limits: Jason Wei’s Framework from Stanford
Meituan Technology Team
Meituan Technology Team
Sep 11, 2025 · Artificial Intelligence

How LongCat-Flash Achieves Ultra-Fast, Low-Cost AI Agent Inference with SGLang

LongCat-Flash, an open‑source Mixture‑of‑Experts model released by Meituan, leverages model‑system co‑design, PD‑disaggregation, SBO scheduling and large‑scale expert parallelism within the SGLang framework to deliver dramatically lower latency, higher throughput and cost‑effective inference for AI agents, with detailed deployment instructions provided.

LongCat-FlashMixture of ExpertsSGLang
0 likes · 15 min read
How LongCat-Flash Achieves Ultra-Fast, Low-Cost AI Agent Inference with SGLang
Data Party THU
Data Party THU
Sep 10, 2025 · Industry Insights

MoE vs MoR: Deep Dive into Expert and Recursive Mixture Architectures for LLMs

This article provides a comprehensive technical comparison between Mixture of Experts (MoE) and the newly proposed Mixture of Recursion (MoR) architectures, covering design principles, parameter efficiency, inference latency, training stability, routing mechanisms, hardware deployment considerations, and suitable application scenarios.

Hardware DeploymentInference PerformanceMixture of Experts
0 likes · 13 min read
MoE vs MoR: Deep Dive into Expert and Recursive Mixture Architectures for LLMs
Data Party THU
Data Party THU
Sep 4, 2025 · Artificial Intelligence

How MXFP4 Quantization Lets a 1200‑Billion‑Parameter LLM Run on a Single 80GB GPU

This article analyzes the memory bottleneck of massive language models, explains the mathematical modeling of memory requirements, evaluates traditional sharding limits, and details how GPT‑OSS’s MXFP4 quantization combined with Mixture‑of‑Experts reduces memory, bandwidth, and compute demands enough to fit a 1200‑billion‑parameter model onto an 80 GB GPU with minimal accuracy loss.

FP4LLMMXFP4
0 likes · 11 min read
How MXFP4 Quantization Lets a 1200‑Billion‑Parameter LLM Run on a Single 80GB GPU
Data Party THU
Data Party THU
Sep 3, 2025 · Artificial Intelligence

Unlocking Large Model Secrets: Transformers, MoE, Fine‑Tuning, RAG & KV Caching

This article provides a comprehensive technical overview of today’s large‑model ecosystem, covering the Transformer architecture, Mixture‑of‑Experts extensions, five fine‑tuning methods, the evolution from traditional RAG to agentic RAG, classic agent design patterns, diverse text‑chunking strategies, and the KV‑cache optimization that accelerates inference.

Fine‑tuningKV cacheMixture of Experts
0 likes · 13 min read
Unlocking Large Model Secrets: Transformers, MoE, Fine‑Tuning, RAG & KV Caching
Baobao Algorithm Notes
Baobao Algorithm Notes
Sep 2, 2025 · Artificial Intelligence

How LongCat‑Flash Achieves Record Speed and Efficiency for a 560B MoE Model

LongCat‑Flash is a 560‑billion‑parameter Mixture‑of‑Experts LLM that combines a dynamic zero‑computation expert design, shortcut‑connected MoE communication, variance‑aligned scaling, and a three‑stage agent‑centric pre‑training pipeline, delivering over 100 TPS on H800 GPUs at a cost of $0.70 per million tokens.

Artificial IntelligenceLongCat-FlashMixture of Experts
0 likes · 23 min read
How LongCat‑Flash Achieves Record Speed and Efficiency for a 560B MoE Model
Bighead's Algorithm Notes
Bighead's Algorithm Notes
Sep 1, 2025 · Artificial Intelligence

How MERA’s Retrieval‑Augmented MoE Boosts Stock Selection Performance by 11%

The article introduces MERA, a Retrieval‑Augmented Mixture‑of‑Experts module that addresses the inability of single‑branch deep‑learning models to capture diverse stock market patterns, describes its self‑supervised pretraining, gating and expert mechanisms, and shows that it improves stock‑selection metrics by up to 11% on major Chinese indices.

MERAMixture of ExpertsRetrieval Augmented Representation
0 likes · 14 min read
How MERA’s Retrieval‑Augmented MoE Boosts Stock Selection Performance by 11%
AI Info Trend
AI Info Trend
Aug 12, 2025 · Artificial Intelligence

OpenAI’s First Open‑Source Weights: Inside gpt‑oss‑120B & 20B Models

OpenAI has unveiled its first open‑source weight models in over five years—gpt‑oss‑120B and gpt‑oss‑20B—detailing their MoE architecture, quantization techniques, benchmark performance, licensing, and the industry’s mixed reactions, while hinting at future open‑source AI developments.

AI benchmarksGPT-OSSMixture of Experts
0 likes · 6 min read
OpenAI’s First Open‑Source Weights: Inside gpt‑oss‑120B & 20B Models
Programmer DD
Programmer DD
Aug 6, 2025 · Artificial Intelligence

What Is GPT-OSS? Inside OpenAI’s New Open‑Source Large Language Models

OpenAI has unveiled GPT‑OSS, an open‑source large language model series featuring a 120‑billion‑parameter version for high‑throughput production and a 20‑billion‑parameter version for low‑latency consumer hardware, both using Mixture‑of‑Experts architecture, 4‑bit quantization, and released under the permissive Apache 2.0 license.

4-bit quantizationApache 2.0 licenseGPT-OSS
0 likes · 3 min read
What Is GPT-OSS? Inside OpenAI’s New Open‑Source Large Language Models