Tagged articles
Sparse Models
13 articles · Page 1 of 1
Qwen researchers propose ID Balancing, a PID-based load control method for extremely sparse MoE models, achieving stable expert load with 768 experts and Top-3 routing, reducing MaxVio by over 50% and MinVio by 12% while maintaining competitive language modeling and downstream task performance.
DeepSeekMoEPID Control
0 likes · 11 min read

This article explains the distinction between total and activated parameters in Mixture-of-Experts (MoE) models, contrasting dense and sparse architectures, detailing expert routing mechanisms, and analyzing memory and compute implications across model loading, prefill, and decode stages.
MoESparse Modelsactivated parameters
0 likes · 15 min read

The article presents STEM, a method that transforms dense and MoE transformer architectures by converting the expert routing step into a static table‑lookup operation, achieving higher parameter efficiency, lower communication overhead, and improved interpretability while maintaining or boosting downstream task performance.
Embedding LookupInterpretabilityModel Efficiency
0 likes · 6 min read

DeepSeek's new paper introduces a conditional memory mechanism called Engram that complements Mixture‑of‑Experts, providing O(1) lookup, improving knowledge retrieval, reasoning, and long‑context performance while scaling efficiently on the same FLOPs budget.
EngramSparse Modelsconditional memory
0 likes · 18 min read

The article analyzes how architecture innovation—through sparse, multimodal, and dynamic designs—will break the compute bottleneck of large models, reshape pre‑training hierarchies, and drive three distinct 2026 pathways for both model efficiency and agent competition.
2026 predictionsAI AgentsSparse Models
0 likes · 12 min read

This article explains the Mixture of Experts (MoE) technique used in modern LLMs, detailing its core components—experts and router—comparing dense and sparse layers, describing load‑balancing, expert capacity, and routing strategies, and showcasing real‑world examples such as Switch Transformer, Vision‑MoE, and Mixtral 8x7B.
Expert CapacityLLMMoE
0 likes · 15 min read

Jeff Dean’s 2024 ETH Zurich talk traces fifteen years of AI breakthroughs—from the rise of neural networks and back‑propagation, through large‑scale distributed training, TPUs, Transformers, sparse MoE models, and advanced prompting techniques—showing how scaling compute, data, and clever software have driven today’s powerful Gemini models.
AIChain-of-ThoughtSparse Models
0 likes · 18 min read


ArchitectMar 2, 2025 · Artificial Intelligence
This article explains the Mixture of Experts (MoE) architecture, detailing experts, routers, dense vs. sparse layers, load‑balancing strategies such as KeepTopK, auxiliary loss, capacity constraints, the Switch Transformer simplification, and how MoE is applied to both language and vision models, illustrated with concrete examples and parameter counts.
MoESparse ModelsSwitch Transformer
0 likes · 17 min read


ArchitectFeb 10, 2025 · Artificial Intelligence
This article reviews the development of DeepSeek's Mixture-of-Experts (MoE) models, tracing their evolution from the original DeepSeekMoE V1 through V2 to V3, detailing architectural innovations such as fine‑grained expert segmentation, shared‑expert isolation, load‑balancing losses, device‑limited routing, and the shift from softmax to sigmoid gating.
DeepSeekLLMSparse Models
0 likes · 21 min read

The article details JD Advertising's technical challenges and solutions for large‑scale sparse recommendation models, describing GPU‑focused storage, compute and I/O optimizations for both training and low‑latency inference, including distributed pipelines, heterogeneous deployment, batch aggregation, multi‑stream execution, and compiler extensions.
GPU optimizationSparse ModelsTensorFlow
0 likes · 13 min read

DeepRec Extension enhances large‑scale sparse model training by adding automatic elastic training, resource‑aware scheduling, real‑time monitoring, and efficient fault‑tolerance mechanisms, enabling lower cost, higher throughput, and more reliable distributed training for AI workloads.
AI infrastructureDeepRecSparse Models
0 likes · 13 min read

The article introduces PaddleBox and FeaBox, two GPU‑driven frameworks designed for massive sparse DNN training and unified feature extraction, detailing their architecture, performance advantages, hardware‑software co‑design challenges, and successful deployment across Baidu's advertising systems.
FeaBoxGPUPaddleBox
0 likes · 24 min read

DeepRec, developed since 2016 by Alibaba, is a specialized sparse‑model training engine that addresses feature elasticity, training performance, and deployment challenges through dynamic elastic features, optimized runtimes, distributed training frameworks, incremental model export, and multi‑level storage, and is now being open‑sourced for broader industry collaboration.
AI infrastructureDeepRecRuntime Optimization
0 likes · 15 min read
