Skip to content
Tagged articles

mixture of experts

201 articles · Page 1 of 3
AI Frontier Lectures
AI Frontier Lectures
Nov 17, 2026 · Artificial Intelligence

Qwen4-Max Cuts Task Cost 62% While Boosting Scores: Architecture & Benchmarks

Tongyi's Qwen4-Max reduces per-task cost from $4.43 to $1.67 (62% less) on its Work360 benchmark while raising scores from 47.74 to 52.69, using a 2.4T-parameter architecture with 710M activated parameters, phrase embeddings, gated recurrent layers, mixture-of-experts, and sparse attention, with training loss declining steadily without spikes.

Qwen4-Maxbenchmarkcompression rate
0 likes · 8 min read
Qwen4-Max Cuts Task Cost 62% While Boosting Scores: Architecture & Benchmarks
SmartApe AI
SmartApe AI
Nov 15, 2026 · Artificial Intelligence

Tencent Hy4 Flash: 12B Active Params, 1/8 DeepSeek Cost, Matches Claude Haiku 5.5

Tencent's Hy4 Flash, a 224B MoE model with 12B active parameters, achieves 150+ tokens/s decoding speed at 1/20 the cost of its predecessor, outperforms DeepSeek-V4.1-Flash on cost efficiency, and matches Claude Haiku 5.5 user satisfaction in blind tests across 259 office tasks.

HunyuanHy4 FlashInference Optimization
0 likes · 6 min read
Tencent Hy4 Flash: 12B Active Params, 1/8 DeepSeek Cost, Matches Claude Haiku 5.5
IT Services Circle
IT Services Circle
Nov 15, 2026 · Artificial Intelligence

7 Trending GitHub Open-Source Projects for AI and Developer Productivity

This roundup covers seven high-quality trending GitHub projects: a categorized public API directory, an AI agent chat manager integrating Claude and Codex, a game studio template with 49 agent roles, offline voice dictation tools, a lightweight headless browser for AI automation, a local MoE model inference engine, and a dictation app with custom vocabulary corrections.

AI agentsGitHubgame development
0 likes · 10 min read
7 Trending GitHub Open-Source Projects for AI and Developer Productivity
PaperAgent
PaperAgent
Nov 14, 2026 · Artificial Intelligence

RSI Closed-Loop Pre-Training Reaches Frontier Scale, Surpasses Qwen3-30B-A3B

The article details the first public case of applying recursive self-improvement (RSI) closed loops to full frontier-scale pre-training, producing the 801-30B-A3B MoE model trained on 11.8T tokens, which outperforms Qwen3-30B-A3B and Nemotron-3.5-Lightning-30B-A3B through three types of feedback loops and agent-generated training data.

RSIagent-generated databenchmarking
0 likes · 9 min read
RSI Closed-Loop Pre-Training Reaches Frontier Scale, Surpasses Qwen3-30B-A3B
SmartApe AI
SmartApe AI
Nov 13, 2026 · Artificial Intelligence

Jeff Dean & Fei-Fei Li: RSI's Physical Limits, Self-Improvement vs. Recursive Self-Improvement

In a Stanford HAI dialogue, Jeff Dean and Fei-Fei Li distinguish self-improvement from recursive self-improvement (RSI), argue RSI has physical limits due to real-world compute constraints, outline Discovery Loop's focus on ML research, engineering, and hardware design, discuss AI's role in autonomous scientific discovery and mathematical verification, and reflect on Transformer, distillation, MoE, and code capabilities as key AI milestones.

Autonomous Scientific DiscoveryDiscovery LoopMathematical Verification
0 likes · 21 min read
Jeff Dean & Fei-Fei Li: RSI's Physical Limits, Self-Improvement vs. Recursive Self-Improvement
SmartApe AI
SmartApe AI
Nov 13, 2026 · Artificial Intelligence

No Second Run: RSI for LLM Pretraining Yields 30B MoE Model in 3 Months

Shanghai AI Lab and Xinxu AI introduce No Second Run, a recursive self-improvement method for LLM pretraining using debugging, headroom evaluation, and experience-to-metric loops. In three months they trained the 801-30B-A3B MoE model (30B total, 3B active params) on 11.8T tokens, scoring 73.13 average on 13 base benchmarks (second only to Qwen3.5-35B-A3B) and achieving top Agent benchmark results after 6.33B-token SFT without RL.

801-30B-A3BAgent BenchmarksLLM Pretraining
0 likes · 15 min read
No Second Run: RSI for LLM Pretraining Yields 30B MoE Model in 3 Months
SmartApe AI
SmartApe AI
Nov 12, 2026 · Artificial Intelligence

ETH Zurich Postdoc Bingcong Li Joins HKU to Drive Efficient LLM Scaling Research

ETH Zurich postdoc Bingcong Li will join HKU as a tenure-track assistant professor, focusing on efficient scaling of large language models through architecture design and optimization, including width/depth scaling, LoRA fine-tuning, MoE efficiency, and zero-order optimization methods like DPZero and RefLoRA.

Architecture DesignEfficient TrainingLoRA
0 likes · 6 min read
ETH Zurich Postdoc Bingcong Li Joins HKU to Drive Efficient LLM Scaling Research
AI Frontier Lectures
AI Frontier Lectures
Nov 11, 2026 · Artificial Intelligence

Praxis-1: Runway's 27B MoE Diffusion Model Leads RoboLab-120 at 46.2% Success

Runway released Praxis-1, a 27B-parameter mixture-of-experts diffusion model that jointly predicts video frames and robot actions, achieving 46.2% success on the RoboLab-120 benchmark over 1,200 trials, outperforming FLUX 3 Action, Cosmos3 Nano, and π0.5, with weights, inference, and fine-tuning code open-sourced under Apache 2.0 and deployable on hardware from H100 to Jetson Thor.

Apache 2.0Praxis-1RoboLab-120
0 likes · 5 min read
Praxis-1: Runway's 27B MoE Diffusion Model Leads RoboLab-120 at 46.2% Success
DataFunTalk
DataFunTalk
Nov 8, 2026 · Artificial Intelligence

DeepSeek-V4-Flash on 64GB CPU: Why 96GB Xeon Can't Load It

The article explains how llama-modchip runs 284B DeepSeek-V4-Flash on a 64GB RAM CPU-only machine via aggressive quantization (1.71 bits/weight), while a 96GB Xeon server fails due to tensor format mismatch and missing AVX-512. It breaks down inference latency for long contracts (~10 minutes for 20k tokens) and proposes three evaluation metrics: time to first token, key-field extraction accuracy, and total processing time.

AVX-512CPU inferenceDeepSeek-V4-Flash
0 likes · 12 min read
DeepSeek-V4-Flash on 64GB CPU: Why 96GB Xeon Can't Load It
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Nov 4, 2026 · Artificial Intelligence

Cortex: Turning Agent Experience into Long-Term Parametric Memory

Researchers from Zhejiang University, ByteDance Seed, and Tsinghua University propose Cortex, a method that consolidates contextual experience into model parameters via prospective rehearsal and context distillation, enabling agents to retain skills and knowledge without external retrieval, validated across tool use, fact updating, and agent tasks with minimal parameter growth.

Continual LearningKnowledge EditingLLM Agents
0 likes · 39 min read
Cortex: Turning Agent Experience into Long-Term Parametric Memory
Su San Talks Tech
Su San Talks Tech
Oct 30, 2026 · Artificial Intelligence

DeepSeek's 10 Game-Changing Open-Source Projects: From GRPO to Agent Infrastructure

This article reviews DeepSeek's 10 most impactful open-source projects, spanning math reasoning (GRPO), MoE communication (DeepEP), distributed storage (3FS), Hopper-optimized kernels (FlashMLA), production inference engine (DeepInfer), optical document compression (DeepSeek-OCR), code generation (Coder-V2), pure RL reasoning (R1), efficient Mixture-of-Experts training (V3), and next-gen agent infrastructure (Harness).

DeepSeekGRPOInference Optimization
0 likes · 25 min read
DeepSeek's 10 Game-Changing Open-Source Projects: From GRPO to Agent Infrastructure
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Oct 26, 2026 · Artificial Intelligence

Datawhale Releases Diy-LLM: Chinese Tutorial for Stanford CS336 with 16 Chapters & 6 Hands-on Assignments

Datawhale community launches Diy-LLM, a Chinese-language tutorial built on Stanford CS336 covering tokenizers, Transformer architecture, MoE, distributed training, scaling laws, inference, alignment, and multimodal models through 16 chapters and 6 practical assignments with code.

DatawhaleLLM trainingStanford CS336
0 likes · 9 min read
Datawhale Releases Diy-LLM: Chinese Tutorial for Stanford CS336 with 16 Chapters & 6 Hands-on Assignments
SmartApe AI
SmartApe AI
Oct 23, 2026 · Artificial Intelligence

IQuest-Q1 Technical Deep Dive: 320B MoE Agent Model with 512K Context

IQuest Research releases technical report for IQuest-Q1, a 320B MoE model with 15B active parameters, 512K context window, recursive multi-token prediction, and agent-focused post-training, achieving strong benchmarks across coding, tool use, and reasoning tasks.

Recursive MTPagent trainingbenchmark evaluation
0 likes · 16 min read
IQuest-Q1 Technical Deep Dive: 320B MoE Agent Model with 512K Context
Machine Heart
Machine Heart
Oct 22, 2026 · Artificial Intelligence

Flint-40B-A2B: AI-Built MoE Coding Model Hits 125 tok/s on MacBook Pro

Recursive_SI's first model Flint-40B-A2B, a 40B MoE model activating 2.5B parameters per token, was built in under four months using the automated Eureka Machine system. It achieves state-of-the-art local coding performance on benchmarks like SWE-Bench while running at 125+ tokens/sec on MacBook Pro M4 Pro and 150+ tokens/sec on NVIDIA RTX Spark, with open weights on Hugging Face under Apache 2.0.

Apple SiliconEureka MachineFlint-40B-A2B
0 likes · 10 min read
Flint-40B-A2B: AI-Built MoE Coding Model Hits 125 tok/s on MacBook Pro
SmartApe AI
SmartApe AI
Oct 21, 2026 · Artificial Intelligence

Recursive Open-Sources Flint-40B-A2B: AI-Designed MoE Model for Local Coding Agents

Recursive team open-sources Flint-40B-A2B, a 40B-parameter MoE model optimized for local coding agents, developed in under four months using their Eureka Machine AI research system; it achieves over 125 tokens/sec on MacBook Pro and demonstrates autonomous research capabilities on tasks like circle packing, factory scheduling (26% improvement), and bike-sharing demand prediction (30.8% RMSLE reduction).

Automated ResearchCoding AgentEureka Machine
0 likes · 6 min read
Recursive Open-Sources Flint-40B-A2B: AI-Designed MoE Model for Local Coding Agents
TechVision Expert Circle
TechVision Expert Circle
Oct 20, 2026 · Artificial Intelligence

Mistral Large 4: 1.05T Open-Source MoE Model Challenges AI Sovereignty

French AI company Mistral releases Mistral Large 4, a 1.05 trillion parameter open-source MoE model with 128 experts and 180B active parameters, achieving 91.2 MMLU-Pro and 89.7% HumanEval+, detailing architecture, training on 16K H200 GPUs, inference via expert offloading, and Apache 2.0 licensing enabling sovereign AI deployment.

1.05T parametersAI sovereigntyMistral Large 4
0 likes · 13 min read
Mistral Large 4: 1.05T Open-Source MoE Model Challenges AI Sovereignty
AntTech
AntTech
Oct 15, 2026 · Artificial Intelligence

Ling-3.1-flash Open-Sourced: 560B MoE Runs on 4 GPUs, 210 tok/s, $0.19/M Tokens

Ant Group's Bailin team open-sources Ling-3.1-flash, a 560B-parameter mixture-of-experts model that activates 25B parameters per token, supports 1M context, and achieves 210 tokens/sec output speed with an intelligence index of 41 on Artificial Analysis, while quantization reduces weight from 534GB to 292GB enabling deployment on 4 Hopper GPUs instead of 8.

GPU InferenceLing-3.1-flashOpen Source LLM
0 likes · 11 min read
Ling-3.1-flash Open-Sourced: 560B MoE Runs on 4 GPUs, 210 tok/s, $0.19/M Tokens
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Oct 11, 2026 · Artificial Intelligence

DeepSeek's Cost Curve Chronicle: How Token Economics Rewrote LLM Architecture

This article chronicles DeepSeek's paper timeline from 2023 to 2026, analyzing how each model generation optimized training compute, KV cache, and throughput via MoE, MLA, sparse attention, and conditional memory, shifting long-context competition from maximum length to per-token byte cost.

DeepSeekGRPOKV cache optimization
0 likes · 37 min read
DeepSeek's Cost Curve Chronicle: How Token Economics Rewrote LLM Architecture
SmartApe AI
SmartApe AI
Oct 8, 2026 · Artificial Intelligence

ID Balancing: PID Control Stabilizes Extremely Sparse MoE with 768 Experts

Qwen researchers propose ID Balancing, a PID-based load control method for extremely sparse MoE models, achieving stable expert load with 768 experts and Top-3 routing, reducing MaxVio by over 50% and MinVio by 12% while maintaining competitive language modeling and downstream task performance.

DeepSeekMoEPID Control
0 likes · 11 min read
ID Balancing: PID Control Stabilizes Extremely Sparse MoE with 768 Experts
Linyb Geek Road
Linyb Geek Road
Sep 29, 2026 · Artificial Intelligence

Reverse-Engineering Jev: 10K API Calls Expose Closed-Source Model Architecture

An independent researcher reverse-engineered TypeSafe's closed-source Jev classification model using 10,000 API calls, revealing its architecture uses a classification head with shared prefix inference, listwise scoring, and exceptional calibration (ECE 0.0313), challenging assumptions about black-box security.

CalibrationJevblack-box probing
0 likes · 22 min read
Reverse-Engineering Jev: 10K API Calls Expose Closed-Source Model Architecture
Machine Heart
Machine Heart
Sep 24, 2026 · Artificial Intelligence

Edge AI Token Factories: Banma Smart's AutoOmni 2.0 Enables Deep AI Usage in Cars

At the 2026 Yunqi Conference, Banma Smart unveiled AutoOmni 2.0-23B-A3B, a sparse MoE edge model that achieves near-cloud performance on vehicle-grade chips, addressing memory, latency, and concurrency constraints while enabling privacy-preserving, low-cost AI with certified safety frameworks for mass production.

AI safetyASPICE CertificationAutoOmni
0 likes · 24 min read
Edge AI Token Factories: Banma Smart's AutoOmni 2.0 Enables Deep AI Usage in Cars
JavaGuide
JavaGuide
Sep 21, 2026 · Artificial Intelligence

Step 5 Preview Outperforms DeepSeek V4 Pro in Coding Tasks at Fraction of Opus 5 Cost

The article benchmarks StepFun's Step 5 Preview against DeepSeek V4 Pro across three coding challenges—2D animation, token bucket demo, and CSV analysis workbench—revealing Step 5 Preview delivers richer details and better test coverage despite longer generation times, with per-task cost at just 35% of GLM-5.3 and 12.5% of Claude Opus 5.

AI coding benchmarkCSV analysisDeepSeek V4 Pro
0 likes · 23 min read
Step 5 Preview Outperforms DeepSeek V4 Pro in Coding Tasks at Fraction of Opus 5 Cost
Data STUDIO
Data STUDIO
Sep 11, 2026 · Artificial Intelligence

Running 744B MoE on 25GB RAM: Int4, Disk Streaming & MLA Compression

This article details how a custom C inference engine runs the 744B-parameter GLM 5.2 MoE model on 25GB RAM by quantizing routed experts to int4, streaming them from disk via pread, compressing KV cache 57x with Multi-head Latent Attention, and employing a five-tier memory hierarchy with speculative decoding.

AVX2 kernelsC inference engineGLM-5.2
0 likes · 48 min read
Running 744B MoE on 25GB RAM: Int4, Disk Streaming & MLA Compression
Machine Heart
Machine Heart
Sep 8, 2026 · Artificial Intelligence

SMELT: Looped Transformers Outperform Baselines Under Fair Budget Matching

The SMELT framework from Tsinghua and ByteDance Seed fairly compares Looped Transformers against baselines by matching compute, parameters, and KV cache, finding that looping the middle 50% of layers twice with a larger depth-width ratio consistently reduces validation loss across scales, saves 6.8–18% training compute, and yields downstream gains beyond loss reduction.

Budget MatchingLLM ArchitectureLooped Transformer
0 likes · 12 min read
SMELT: Looped Transformers Outperform Baselines Under Fair Budget Matching
Architects' Tech Alliance
Architects' Tech Alliance
Sep 6, 2026 · Artificial Intelligence

xDeepServe on CloudMatrix384: Full-Stack Design for Large-Scale MoE Model Serving

This article details Huawei's xDeepServe system for deploying massive Mixture-of-Experts models on the CloudMatrix384 supernode, covering the XCCL communication library, FlowServe decentralized serving engine, Transformerless execution architecture with prefill-decode and MoE-Attention decoupling, and hierarchical fault tolerance, achieving 2400 tokens/s per chip at 50ms TPOT.

Ascend 910CCloudMatrix384EPLB
0 likes · 18 min read
xDeepServe on CloudMatrix384: Full-Stack Design for Large-Scale MoE Model Serving
Xiaomi Tech
Xiaomi Tech
Sep 5, 2026 · Artificial Intelligence

Xiaomi-TabLDM: One Model for All Tabular Tasks via Synthetic Pretraining & Test-Time Scaling

Xiaomi releases Xiaomi-TabLDM, a foundation model for tabular data that uses large-scale synthetic pretraining, efficient model scaling with dual-stream feature groups and sparse MoE, and test-time scaling to achieve top-tier performance on four public benchmarks and real-world industrial tasks without per-dataset retraining.

Test-Time ScalingXiaomi-TabLDMbenchmark
0 likes · 9 min read
Xiaomi-TabLDM: One Model for All Tabular Tasks via Synthetic Pretraining & Test-Time Scaling
Cambridge Mofang Notes
Cambridge Mofang Notes
Sep 1, 2026 · Artificial Intelligence

From Dense to MoE: Decoding Total vs. Activated Parameters

This article explains the distinction between total and activated parameters in Mixture-of-Experts (MoE) models, contrasting dense and sparse architectures, detailing expert routing mechanisms, and analyzing memory and compute implications across model loading, prefill, and decode stages.

MoESparse Modelsactivated parameters
0 likes · 15 min read
From Dense to MoE: Decoding Total vs. Activated Parameters
Tencent Technical Engineering
Tencent Technical Engineering
Aug 28, 2026 · Artificial Intelligence

Tencent Hunyuan Hy4 Preview: 770B Open-Source Model Claims Top Tier

Tencent releases Hunyuan Hy4 preview, a 770B parameter mixture-of-experts model with 49B activated parameters and 1M context length, achieving top-tier open-source performance across coding, office, gaming, and scientific tasks, with internal blind tests showing 2.99/4 score surpassing GLM 5.3 and Kimi K3, plus 31.8% inference throughput gains via self-optimization.

HunyuanHy4Inference Optimization
0 likes · 6 min read
Tencent Hunyuan Hy4 Preview: 770B Open-Source Model Claims Top Tier
SuanNi
SuanNi
Aug 21, 2026 · Artificial Intelligence

Ornith-1.5 Hits SOTA 9B/35B and Matches Claude Opus 4.8 at 397B

Ornith-1.5, an MIT‑licensed large‑model framework from DeepReinforce, introduces a self‑improving loop that autonomously generates tasks, builds scaffolds, and rolls out solutions, achieving state‑of‑the‑art performance at 9B and 35B scales and delivering benchmark scores comparable to Claude Opus 4.8 for its 397B MoE variant.

AIOrnith-1.5Self-Improvement
0 likes · 7 min read
Ornith-1.5 Hits SOTA 9B/35B and Matches Claude Opus 4.8 at 397B
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Aug 17, 2026 · Artificial Intelligence

N Load‑Balancing Techniques for Large‑Scale MoE Models

The article explains why token‑to‑expert routing in large Mixture‑of‑Experts models can become highly imbalanced, outlines the resulting compute bottlenecks, token overflow and expert degradation, and then surveys eight concrete load‑balancing strategies—ranging from importance‑based auxiliary losses to capacity‑aware token dropping and loss‑free bias control—detailing their mechanisms, trade‑offs, and practical considerations.

Auxiliary LossBalanced AssignmentCapacity Factor
0 likes · 17 min read
N Load‑Balancing Techniques for Large‑Scale MoE Models
TechVision Expert Circle
TechVision Expert Circle
Aug 17, 2026 · Artificial Intelligence

Open-Source LLMs Close the Gap: Low-Cost AI Poised to Redefine the Market

In early 2026, open-weight LLMs such as Llama 4, Qwen 3 and DeepSeek‑V3/R2 began matching or surpassing leading closed models like GPT‑4.5 and Claude Opus 5, driven by MoE architectures, FP8 precision, GRPO training and aggressive inference optimizations, prompting a reassessment of enterprise AI strategy.

AI economicsInference OptimizationOpen Source LLM
0 likes · 15 min read
Open-Source LLMs Close the Gap: Low-Cost AI Poised to Redefine the Market
Old Zhang's AI Learning
Old Zhang's AI Learning
Aug 17, 2026 · Artificial Intelligence

How Many GPUs Does Kimi K3 Need? Self‑Hosting vs API Cost Comparison

The article breaks down Kimi K3’s 2.8‑trillion‑parameter architecture, explains its 4‑bit MXFP4 quantization, calculates the ~1.4 TB memory requirement, shows that 8‑GPU clusters (e.g., NVIDIA B300 or AMD MI350X) are needed for self‑hosting, and compares these costs with the per‑token API pricing, highlighting when each option is economical.

API costDigitalOceanGPU requirements
0 likes · 14 min read
How Many GPUs Does Kimi K3 Need? Self‑Hosting vs API Cost Comparison
21CTO
21CTO
Aug 9, 2026 · Artificial Intelligence

ByteDance’s 10‑Trillion‑Parameter Gamble: Inside China’s Largest AI Model Pre‑training

ByteDance is reportedly pre‑training a 10‑trillion‑parameter AI model that dwarfs domestic rivals, uses a Mixture‑of‑Experts architecture with low activation ratios, demands roughly 36 000 Blackwell GPUs and $2.5 billion in hardware, and deliberately avoids distilling competitor models, raising questions about China’s AI frontier timeline.

AI scalingByteDanceDistillation avoidance
0 likes · 5 min read
ByteDance’s 10‑Trillion‑Parameter Gamble: Inside China’s Largest AI Model Pre‑training
21CTO
21CTO
Aug 8, 2026 · Artificial Intelligence

Jeff Dean Discusses the Next Decade of AI Just 12 Hours After Leaving Google

In a Stanford interview hosted by Dawn Song, Jeff Dean reflects on the origins of MoE, the transformative impact of deep learning, lessons from TensorFlow, how to spot breakthrough directions, AI agent risks, and his new venture Discovery Loop that aims to automate scientific research.

AI safetyDiscovery LoopJeff Dean
0 likes · 12 min read
Jeff Dean Discusses the Next Decade of AI Just 12 Hours After Leaving Google
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Aug 5, 2026 · Artificial Intelligence

Inside K3: How Stable Latent MoE and MLA Attention Are Designed

The article examines K3’s architecture—combining KDA, MLA, Stable Latent MoE and AttnRes—detailing the replacement of SwiGLU with SiTU‑GLU, the addition of RMSNorm for training stability, the Quantile Balancing load‑balancing scheme, and the trade‑offs behind its MLA and NoPE attention choices.

K3MLA AttentionQuantile Balancing
0 likes · 15 min read
Inside K3: How Stable Latent MoE and MLA Attention Are Designed
21CTO
21CTO
Aug 3, 2026 · Artificial Intelligence

JetBrains’ Open‑Source 12B Code Model Mellum2: Private Deployment and High‑Throughput Over Claude Code

JetBrains released the open‑source 12‑billion‑parameter Mellum2 model, a MoE‑based code AI that delivers private on‑prem deployment, high‑throughput inference, and strong code‑generation benchmarks, positioning it as a fast, specialized alternative to Claude Code and other proprietary models.

Mellum2benchmarkcode generation
0 likes · 7 min read
JetBrains’ Open‑Source 12B Code Model Mellum2: Private Deployment and High‑Throughput Over Claude Code
Black & White Path
Black & White Path
Aug 2, 2026 · Artificial Intelligence

Running a 2.8‑Trillion‑Parameter K3 Model on 4 GB VRAM with AirLLM

AirLLM introduces layer‑wise inference and per‑expert streaming to decouple VRAM usage from model size, enabling the 2.8‑trillion‑parameter Kimi K3 LLM to run on a single consumer‑grade GPU while preserving full‑precision accuracy and offering security‑focused insights.

AirLLMKimi K3Layer-wise Inference
0 likes · 9 min read
Running a 2.8‑Trillion‑Parameter K3 Model on 4 GB VRAM with AirLLM
ITPUB
ITPUB
Jul 28, 2026 · Artificial Intelligence

Why Kimi K3’s Open‑Source Release Puts China at the Forefront of Global AI

Kimi K3, a 2.8‑trillion‑parameter MoE model with a 100 k‑token context, has been fully open‑sourced along with its weights, technical report and infra (MoonEP, FlashKDA, AgentEnv), delivering programming and agent benchmark results that rival top closed models such as Claude Fable 5 and GPT‑5.6 while sparking debate over alleged distillation and emphasizing AI safety and open‑weight governance.

AI benchmarksAI safetyKimi K3
0 likes · 11 min read
Why Kimi K3’s Open‑Source Release Puts China at the Forefront of Global AI
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Jul 27, 2026 · Artificial Intelligence

Scaling Residual Streams Efficiently: From DeepSeek mHC to xHC’s 16‑Stream Expansion

The blog details how xHC expands language‑model residual streams to 16, achieving nearly double the gain of DeepSeek mHC on 18B and 28B MoE models, and explains the design of Temporal Feature Augmentation and Sparse Write that make large‑N scaling both effective and affordable.

Hyper-Connectionslarge language modelsmixture of experts
0 likes · 24 min read
Scaling Residual Streams Efficiently: From DeepSeek mHC to xHC’s 16‑Stream Expansion
AI Programming Lab
AI Programming Lab
Jul 21, 2026 · Artificial Intelligence

How Kimi K3 and Qwen3.8‑Max Reach 2T+ Parameters: The Evolution of Chinese LLM Architecture

The article examines how Chinese LLMs such as Kimi K3 (2.8 T) and Qwen 3.8‑Max (2.4 T) achieved 2‑trillion‑parameter scales by progressively decoupling parameters from compute with Mixture‑of‑Experts, optimizing attention, and introducing training tricks like Muon and low‑precision quantization, tracing four years of architectural advances.

Model Scalingattention mechanismslarge language models
0 likes · 11 min read
How Kimi K3 and Qwen3.8‑Max Reach 2T+ Parameters: The Evolution of Chinese LLM Architecture
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Jul 19, 2026 · Artificial Intelligence

How Kimi K3 Highlights Latent MoE as the Next Turning Point in Mixture‑of‑Experts Architecture

Latent MoE, demonstrated by NVIDIA’s Nemotron 3 Super and Moonshot AI’s 2.8 T‑parameter Kimi K3, compresses expert computations into a lower‑dimensional latent space, cutting memory reads and All‑to‑All traffic by fourfold, enabling more experts per token, higher accuracy, and up to 3.5× faster inference.

AI model scalingKimi K3Latent MoE
0 likes · 10 min read
How Kimi K3 Highlights Latent MoE as the Next Turning Point in Mixture‑of‑Experts Architecture
21CTO
21CTO
Jul 17, 2026 · Artificial Intelligence

Kimi K3 Unveiled: 2.8 Trillion‑Parameter Open‑Source LLM Sets New Record

On July 16, the Moon‑of‑Darkness team released Kimi K3, a 2.8‑trillion‑parameter open‑source large language model that introduces mixed‑linear attention, attention residuals, and a highly efficient Mixture‑of‑Experts design, achieving roughly 2.5× the scaling efficiency of its predecessor while approaching the performance of top closed‑source models.

Kimi K3large language modelmixed linear attention
0 likes · 6 min read
Kimi K3 Unveiled: 2.8 Trillion‑Parameter Open‑Source LLM Sets New Record
SuanNi
SuanNi
Jul 17, 2026 · Artificial Intelligence

Kimi K3: The World’s First 3‑Trillion‑Parameter Open‑Source Model

Kimi K3, a 2.8‑trillion‑parameter open‑source LLM, outperforms top closed‑source models in benchmarks, excels at long‑range coding, GPU kernel optimization, and multimodal tasks, while introducing novel attention mechanisms, a compact Triton‑like compiler, and even a prototype ASIC chip.

GPU compilationKimi K3Open Source LLM
0 likes · 9 min read
Kimi K3: The World’s First 3‑Trillion‑Parameter Open‑Source Model
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Jul 16, 2026 · Artificial Intelligence

Inkling – A 975‑Billion‑Parameter Open‑Weight Multimodal Model from Thinking Machines

Thinking Machines Lab unveiled Inkling, a 975‑billion‑parameter open‑weight multimodal model featuring a hybrid‑expert Transformer, 1‑million‑token context, and extensive benchmark results, alongside the lighter Inkling‑Small, with detailed architecture, training methodology, reinforcement‑learning enhancements, and practical examples of web‑app generation and tool‑calling.

Inklingbenchmarklarge language model
0 likes · 15 min read
Inkling – A 975‑Billion‑Parameter Open‑Weight Multimodal Model from Thinking Machines
TechVision Expert Circle
TechVision Expert Circle
Jul 16, 2026 · Artificial Intelligence

Enterprise AI Trends for H2 2026: Key Priorities for Tech Leaders

In the second half of 2026, enterprise AI shifts from adoption to reliable, cost‑effective deployment, with six key trends—including multi‑agent orchestration, GraphRAG retrieval, MoE model clusters, AI observability, built‑in data governance, and reorganized AI engineering roles—guiding tech leaders toward trustworthy AI systems.

AI Team StructureAI agentAI observability
0 likes · 13 min read
Enterprise AI Trends for H2 2026: Key Priorities for Tech Leaders
Architecture Digest
Architecture Digest
Jul 14, 2026 · Artificial Intelligence

How DeepSeek V4’s CSA+HCA Architecture Turns Million‑Token Context into a Cheap Feature

DeepSeek V4 introduces a hybrid CSA/HCA attention mechanism and a Mixture‑of‑Experts design that reduces inference cost to 27% of its predecessor, enables a 1‑million‑token context as a standard feature, and runs efficiently on both NVIDIA GPUs and Huawei Ascend NPU, dramatically lowering the barrier for Java developers to integrate large‑scale AI.

CSAHCAattention compression
0 likes · 10 min read
How DeepSeek V4’s CSA+HCA Architecture Turns Million‑Token Context into a Cheap Feature
Black & White Path
Black & White Path
Jul 14, 2026 · Artificial Intelligence

SuperGemma 26B: The Fully Uncensored ‘Zero‑Guardrails’ AI Model Explained

Independent developer David Ondrej released SuperGemma 26B, an uncensored fork of Google’s Gemma 4 26B that removes all safety guardrails, runs locally on consumer‑grade GPUs, and has sparked intense debate over its technical merits, deployment simplicity, and the security risks of a truly unrestricted AI model.

AI safetyGemma 4Open Source Model
0 likes · 11 min read
SuperGemma 26B: The Fully Uncensored ‘Zero‑Guardrails’ AI Model Explained
Geek Labs
Geek Labs
Jul 14, 2026 · Artificial Intelligence

How Colibri Runs a 744 B MoE Model on CPU with Only 25 GB RAM

Colibri demonstrates that the 744‑billion‑parameter GLM‑5.2 MoE model can be executed on a pure‑CPU system using roughly 25 GB of memory by streaming expert weights from disk, int4‑quantizing dense parameters, and employing a compact C inference engine.

CPU inferenceGLM-5.2INT4 quantization
0 likes · 7 min read
How Colibri Runs a 744 B MoE Model on CPU with Only 25 GB RAM
Machine Heart
Machine Heart
Jul 13, 2026 · Artificial Intelligence

Breaking the Forgetting Barrier: CaRE Scales Continual Learning to 300+ Tasks

The paper introduces CaRE, a scalable continual‑learning framework that leverages a bi‑level routing mixture‑of‑experts to successfully train Vision Transformers on over 300 non‑overlapping tasks, outperforming existing baselines and accompanied by a new 1,000‑class benchmark, OmniBenchmark‑1K.

Bi-Level RoutingCaREContinual Learning
0 likes · 11 min read
Breaking the Forgetting Barrier: CaRE Scales Continual Learning to 300+ Tasks
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Jul 10, 2026 · Artificial Intelligence

World’s First Embodied‑Native Action Model: Inside LingBot‑VA 2.0

LingBot‑VA 2.0 introduces the industry’s first embodied‑native pre‑trained robot brain, combining causal action modeling, a sparse MoE backbone, a semantic VAE tokenizer and asynchronous foresight reasoning to achieve six‑fold inference speedup, single‑GPU deployment and a 93.6% success rate on the RoboTwin 2.0 benchmark.

Causal ModelingEmbodied AIForesight Reasoning
0 likes · 11 min read
World’s First Embodied‑Native Action Model: Inside LingBot‑VA 2.0
Machine Heart
Machine Heart
Jul 9, 2026 · Artificial Intelligence

LingBot-Video: The Open‑Source Video Foundation Model Tailored for Robots

LingBot-Video is an open‑source video‑generation foundation model built for embodied AI, combining a sparse‑Mixture‑of‑Experts architecture, a multi‑stage data curriculum and six‑dimensional reward learning to achieve physically consistent video synthesis that outperforms existing open‑source baselines in both visual quality and robotic relevance.

Embodied AIMultimodal Modeldata curriculum
0 likes · 22 min read
LingBot-Video: The Open‑Source Video Foundation Model Tailored for Robots
Machine Heart
Machine Heart
Jul 8, 2026 · Artificial Intelligence

How LingBot‑VLA 2.0 Powers 20 Robot Configurations with an Open‑Source Embodied Brain

LingBot‑VLA 2.0 introduces a token‑level loss‑free MoE, dual‑query distillation, and a 60k‑hour heterogeneous dataset to achieve cross‑embodiment visual‑language‑action capabilities across 20 robot morphologies, delivering superior benchmark performance and sub‑130 ms inference while being fully open‑sourced.

Data EngineeringEmbodied AIFuture Prediction
0 likes · 16 min read
How LingBot‑VLA 2.0 Powers 20 Robot Configurations with an Open‑Source Embodied Brain
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Jul 1, 2026 · Artificial Intelligence

SAME: Stabilizing MoE to Reduce Dual Forgetting in Multimodal Continual Instruction Tuning

The paper identifies routing drift and expert drift as the two main causes of forgetting in multimodal continual instruction tuning (MCIT) and proposes SAME, which combines spectral‑aware routing, curvature‑aware scaling, and adaptive expert activation to keep MoE models stable, efficient, and less forgetful across long task sequences.

Continual LearningICML 2026Same
0 likes · 19 min read
SAME: Stabilizing MoE to Reduce Dual Forgetting in Multimodal Continual Instruction Tuning
DeepHub IMBA
DeepHub IMBA
Jun 22, 2026 · Artificial Intelligence

How DiffusionGemma Shifts LLM Inference Bottleneck from Memory Bandwidth to Compute

DiffusionGemma, an experimental discrete text diffusion model built on the 26B MoE Gemma‑4 architecture, generates whole 256‑token blocks with bidirectional attention, moving the inference bottleneck from memory bandwidth to GPU compute, achieving up to four‑fold speed gains on H100 and RTX 5090 GPUs, though with lower output quality than standard autoregressive models.

DiffusionGemmaGPU performanceLLM inference
0 likes · 7 min read
How DiffusionGemma Shifts LLM Inference Bottleneck from Memory Bandwidth to Compute
AI Engineer Programming
AI Engineer Programming
Jun 17, 2026 · Artificial Intelligence

Local LLMs Viable: Sparse Attention, MoE, KV Compression, Multi‑Token Prediction

In early 2026, open‑source local large language models become practical alternatives thanks to sparse attention, MoE routing, latent KV compression, multi‑token prediction, and 4‑bit quantization, while hardware memory shortages and benchmark gaps with closed‑source models shape their deployment choices.

4-bit quantizationKV compressionMulti-Token Prediction
0 likes · 13 min read
Local LLMs Viable: Sparse Attention, MoE, KV Compression, Multi‑Token Prediction
AI Insight Log
AI Insight Log
Jun 12, 2026 · Artificial Intelligence

Kimi K2.7 Code: 1T MoE Model Cuts Tokens 30% and Beats Claude Opus on MCP Calls

The newly released Kimi K2.7 Code, a 1‑trillion‑parameter mixture‑of‑experts model that activates only 32 B parameters per inference, offers a 256 K context window, supports multimodal input, improves benchmark scores by up to 31.5 % over K2.6, reduces inference token usage by about 30 %, and achieves an 81.1 MCP tool‑call score surpassing Claude Opus 4.8, while providing a CLI installation command and usage guidelines.

Coding ModelKimiMCP
0 likes · 7 min read
Kimi K2.7 Code: 1T MoE Model Cuts Tokens 30% and Beats Claude Opus on MCP Calls
HyperAI Super Neural
HyperAI Super Neural
Jun 12, 2026 · Artificial Intelligence

DiffusionGemma Boosts Text Generation Speed Up to 4× with Discrete Diffusion

Google’s open‑source DiffusionGemma model leverages a 26‑billion‑parameter Mixture‑of‑Experts architecture and discrete diffusion decoding to generate whole text blocks, achieving up to four times faster generation—over 1100 tokens/s on an NVIDIA H100 and 700 tokens/s on an RTX 5090—while activating only 3.8 billion parameters during inference.

DiffusionGemmaDiscrete DiffusionGPU acceleration
0 likes · 4 min read
DiffusionGemma Boosts Text Generation Speed Up to 4× with Discrete Diffusion
Machine Heart
Machine Heart
Jun 11, 2026 · Artificial Intelligence

Google Releases DiffusionGemma 26B MoE—Text Generation Up to 4× Faster

DiffusionGemma, Google's new 26‑billion‑parameter Mixture‑of‑Experts model, replaces token‑by‑token autoregression with a diffusion‑style output head that generates whole text blocks, delivering up to four‑fold speed gains on consumer GPUs while offering bidirectional attention and self‑correction, albeit with lower quality than standard Gemma 4.

DiffusionGemmaGPU accelerationbidirectional attention
0 likes · 6 min read
Google Releases DiffusionGemma 26B MoE—Text Generation Up to 4× Faster
Data Party THU
Data Party THU
Jun 6, 2026 · Artificial Intelligence

How a 400B MoE Model Runs on iPhone 17 Pro with Flash‑MoE

The article details how the open‑source Flash‑MoE engine enables the 400B‑parameter Qwen3.5‑397B‑A17B mixture‑of‑experts model to run on an iPhone 17 Pro, achieving about 0.6 tokens per second through a custom Metal pipeline, GCD‑driven SSD streaming, and aggressive caching strategies.

400BFlash-MoELLM inference
0 likes · 6 min read
How a 400B MoE Model Runs on iPhone 17 Pro with Flash‑MoE
Data Party THU
Data Party THU
May 29, 2026 · Artificial Intelligence

Token Superposition Training: 2.5× Faster LLM Pre‑training Without Model Changes

The article presents Token Superposition Training (TST), which temporarily averages embeddings of non‑overlapping token bags and predicts groups of tokens in a first phase before reverting to standard token‑wise prediction, achieving up to 2.5× pre‑training speedup on 10B‑1B MoE models without altering model architecture or inference.

LLM PretrainingMCE lossMulti-Token Prediction
0 likes · 9 min read
Token Superposition Training: 2.5× Faster LLM Pre‑training Without Model Changes
Architect's Guide
Architect's Guide
May 29, 2026 · Artificial Intelligence

What Makes DeepSeek V4 Different? A Deep Technical Dive into Its Innovations

DeepSeek V4 introduces a suite of architectural breakthroughs—including mixed‑expert MoE, manifold‑constrained hyper‑connections, CSA/HCA hybrid attention, and FP4 quantization—that slash inference cost by up to tenfold while delivering million‑token context, competitive benchmarks, dual model variants, and a disruptive pricing strategy.

AI model benchmarkAgentic AIFP4 quantization
0 likes · 41 min read
What Makes DeepSeek V4 Different? A Deep Technical Dive into Its Innovations
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
May 22, 2026 · Artificial Intelligence

20‑Year‑Old Transformer Co‑author Open‑Sources a 218‑Billion‑Parameter Model

Cohere’s Command A+ model, built by Transformer co‑author Aidan Gomez and backed by Nick Frosst, packs 218 billion parameters but activates only 25 billion at inference, uses a lossless 4‑bit quantization scheme, offers native citation support, runs on a single B200 or two H100 GPUs, and is released under an Apache 2.0 license, marking a major shift toward truly open‑source, enterprise‑ready large language models.

AIApache 2.0Cohere
0 likes · 12 min read
20‑Year‑Old Transformer Co‑author Open‑Sources a 218‑Billion‑Parameter Model
Machine Heart
Machine Heart
May 22, 2026 · Artificial Intelligence

Breaking the Echo Chamber: MP‑MoE Introduces Ensemble‑Pruning for Diverse Experts

The paper presents MP‑MoE, a new Mixture‑of‑Experts architecture that replaces top‑k routing with Mahalanobis‑based ensemble pruning, explicitly encouraging expert diversity via a co‑occurrence matrix, and uses an efficient greedy algorithm with incremental Cholesky updates, achieving higher performance with minimal training overhead and no inference cost.

Expert DiversityICML 2026MP-MoE
0 likes · 8 min read
Breaking the Echo Chamber: MP‑MoE Introduces Ensemble‑Pruning for Diverse Experts
Alibaba International Intelligent Technology
Alibaba International Intelligent Technology
May 22, 2026 · Artificial Intelligence

How SORT Transforms Precision Ranking with a Transformer‑Based Architecture

SORT re‑architects industrial‑scale ranking by shifting to a request‑centric data paradigm, integrating sparse and MoE optimizations into a Transformer backbone, and delivering significant CTR‑AUC, FLOPs, and online metric improvements while maintaining high training and inference efficiency.

Large Model ScalingRankingTransformer
0 likes · 19 min read
How SORT Transforms Precision Ranking with a Transformer‑Based Architecture
Machine Heart
Machine Heart
May 20, 2026 · Artificial Intelligence

Can Tabular Anomaly Detection Move Beyond One‑for‑One? OFA‑TAD Introduces a One‑for‑All Paradigm

Tabular anomaly detection traditionally requires training a separate model for each dataset (one‑for‑one), but the new OFA‑TAD framework trains once on multiple source tables and directly transfers to unseen target tables without fine‑tuning, leveraging multi‑view distance encoding, MoE fusion, and synthetic pseudo‑anomalies to achieve state‑of‑the‑art performance across 34 datasets in 14 domains.

OFA-TADmixture of expertsmulti-view distance
0 likes · 10 min read
Can Tabular Anomaly Detection Move Beyond One‑for‑One? OFA‑TAD Introduces a One‑for‑All Paradigm
Data Party THU
Data Party THU
May 17, 2026 · Artificial Intelligence

How DeepSeek Leverages MoE Parallelism: GPU Compute and Communication Optimizations

The article dissects DeepSeek's MoE model‑parallel strategy, explaining how GPU compute and communication are overlapped through expert, pipeline, and ZeRO‑1 parallelism, and introduces DualPipe and Waved‑EP kernels that enable efficient training on large‑scale hardware.

DeepSeekGPU Communication OverlapPipeline Parallelism
0 likes · 18 min read
How DeepSeek Leverages MoE Parallelism: GPU Compute and Communication Optimizations
PaperAgent
PaperAgent
May 13, 2026 · Artificial Intelligence

One-for-All Multi-Agent Collaboration: Adaptive Cross-Task Topology Design

The paper introduces OFA-MAS, a one‑for‑all multi‑agent system that learns a universal topology designer using task‑aware graph encoding and a Mixture‑of‑Experts generator, achieving superior performance, OOD generalization, robustness, and efficiency across six major benchmarks.

LLMTask-Aware Graph EncoderTopology Design
0 likes · 14 min read
One-for-All Multi-Agent Collaboration: Adaptive Cross-Task Topology Design
Lao Guo's Learning Space
Lao Guo's Learning Space
May 12, 2026 · Artificial Intelligence

Demystifying the Core Technologies Behind ChatGPT, GPT‑4, and DeepSeek

This article breaks down the key algorithms that power large‑language models—Transformer, Mixture‑of‑Experts, Flash Attention, KV‑Cache, Multi‑Token Prediction, quantization, Chain‑of‑Thought and Retrieval‑Augmented Generation—explaining how each contributes to the performance of ChatGPT, GPT‑4 and DeepSeek.

Flash AttentionKV CacheRetrieval-Augmented Generation
0 likes · 10 min read
Demystifying the Core Technologies Behind ChatGPT, GPT‑4, and DeepSeek
Old Zhang's AI Learning
Old Zhang's AI Learning
May 11, 2026 · Artificial Intelligence

Open‑Source Qwen3.6‑35B‑A3B Runs at 162 tok/s on a Single RTX 5090

The article introduces the open‑source Qwen3.6‑35B‑A3B model, explains its MoE architecture, three‑stage LoRA fine‑tuning, shows benchmark results where it achieves 161.9 tok/s on an RTX 5090—2.6× faster than a dense 27B counterpart—and discusses deployment tips, quantized GGUF release, and known compatibility pitfalls.

GGUF quantizationLoRA fine-tuningQwen3.6-35B-A3B
0 likes · 7 min read
Open‑Source Qwen3.6‑35B‑A3B Runs at 162 tok/s on a Single RTX 5090
Old Zhang's AI Learning
Old Zhang's AI Learning
May 7, 2026 · Artificial Intelligence

How Unsloth and NVIDIA Boost Consumer‑GPU LLM Training by ~25% with Three Simple Optimizations

Unsloth and NVIDIA identified three low‑level bottlenecks in LLM fine‑tuning on consumer GPUs—repeated packed‑sequence metadata construction, serialized copy‑and‑compute during gradient checkpointing, and per‑expert routing overhead in MoE—and applied targeted patches that together deliver roughly a 25% speedup without changing hardware, code, or frameworks.

GPU optimizationGradient CheckpointingLLM training
0 likes · 12 min read
How Unsloth and NVIDIA Boost Consumer‑GPU LLM Training by ~25% with Three Simple Optimizations
AI Engineer Programming
AI Engineer Programming
May 7, 2026 · Artificial Intelligence

How Cursor Turned Its Coding Agent from Demo to Production

The article examines Cursor's journey of shipping its Composer coding agent, detailing the agentic AI model, system architecture, and the three major production challenges—diff handling, latency accumulation, and sandbox scaling—along with the engineering solutions that enabled reliable, fast, and adoptable AI‑driven code generation.

Agentic AICoding AgentCursor
0 likes · 16 min read
How Cursor Turned Its Coding Agent from Demo to Production
Machine Heart
Machine Heart
May 4, 2026 · Artificial Intelligence

Mega MoE vs SonicMoE: Which Will Lead the Next AI Speed Race?

SonicMoE, a new ultra‑fast Mixture‑of‑Experts model from Tri Dao and Ion Stoica’s team, achieves peak throughput on Nvidia Blackwell GPUs, outperforms DeepSeek’s DeepGEMM, and introduces algorithmic redesigns that decouple activation memory from expert granularity while fusing I/O‑aware kernels for up to double the speed of existing MoE frameworks.

AI performanceBlackwellGPU acceleration
0 likes · 12 min read
Mega MoE vs SonicMoE: Which Will Lead the Next AI Speed Race?
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
May 3, 2026 · Artificial Intelligence

Running a 400B Mixture‑of‑Experts LLM on iPhone 17 Pro: Inside Flash‑MoE

The article details how the open‑source Flash‑MoE engine streams a 400‑billion‑parameter Mixture‑of‑Experts language model on an iPhone 17 Pro, achieving interactive‑level token throughput by eliminating Python dependencies, crafting a custom Metal pipeline, and streaming weights directly from SSD.

Apple SiliconFlash-MoEGCD
0 likes · 7 min read
Running a 400B Mixture‑of‑Experts LLM on iPhone 17 Pro: Inside Flash‑MoE
Data Party THU
Data Party THU
May 2, 2026 · Artificial Intelligence

Training an 11.5 B‑parameter Universal Interatomic Potential in Hours on Exascale Supercomputers

A Chinese Academy of Sciences team introduced the MatRIS‑MoE model and the Janus training framework, enabling a 11.5 billion‑parameter universal machine‑learning interatomic potential to be trained on two exascale systems at 1.2 EFLOPS, compressing weeks‑long training into a few hours.

AI for ScienceExascale trainingHigh-Performance Computing
0 likes · 8 min read
Training an 11.5 B‑parameter Universal Interatomic Potential in Hours on Exascale Supercomputers
Machine Heart
Machine Heart
May 1, 2026 · Artificial Intelligence

How a 400B Mixture‑of‑Experts Model Runs on the iPhone 17 Pro

The article details the Flash‑MoE project that streams the 400 billion‑parameter Qwen3.5‑397B‑A17B mixture‑of‑experts model on an iPhone 17 Pro, achieving up to 0.6 tokens per second with a custom Metal‑GPU pipeline, zero‑Python code, and SSD‑backed weight streaming that keeps only 5.5 GB in RAM.

Flash-MoELLMMetal
0 likes · 7 min read
How a 400B Mixture‑of‑Experts Model Runs on the iPhone 17 Pro
Machine Heart
Machine Heart
Apr 30, 2026 · Artificial Intelligence

Beyond DeepSeek V4: A Trillion‑Parameter LLM Trained End‑to‑End on Domestic Chips

The article analyzes how both DeepSeek V4 and Meituan's LongCat‑2.0‑P preview, each with trillion‑scale parameters and 1 M‑token context, were trained and inferred entirely on Chinese‑made accelerators, detailing memory optimizations, deterministic operators, MoE redesigns, and massive multi‑card clusters that prove domestic compute can meet top‑tier AI workloads.

Deterministic OpsDomestic AI ChipLongCat
0 likes · 13 min read
Beyond DeepSeek V4: A Trillion‑Parameter LLM Trained End‑to‑End on Domestic Chips
Architect's Must-Have
Architect's Must-Have
Apr 28, 2026 · Artificial Intelligence

Why DeepSeek V4 Stands Apart: A Deep Dive into Its Architecture and Performance

DeepSeek V4 introduces a suite of architectural innovations—including mixed attention, manifold‑constrained hyper‑connections, the Muon optimizer, and FP4‑aware quantization—that together slash million‑token inference cost to a tenth of its predecessor while delivering benchmark results that rival top‑tier closed‑source models.

EfficiencyFP4 quantizationMuon optimizer
0 likes · 44 min read
Why DeepSeek V4 Stands Apart: A Deep Dive into Its Architecture and Performance
CodeTrend
CodeTrend
Apr 26, 2026 · Artificial Intelligence

DeepSeek V4 Architecture: High‑Efficiency Long‑Context Model Design

DeepSeek V4, released in April 2026, introduces two versions—Pro and Flash—with up to 1.6 trillion parameters and a million‑token context window, leveraging hybrid attention, compressed KV cache, and specialized training techniques to dramatically cut hardware dependence and inference cost.

DeepSeekFP4V4
0 likes · 5 min read
DeepSeek V4 Architecture: High‑Efficiency Long‑Context Model Design
AI2ML AI to Machine Learning
AI2ML AI to Machine Learning
Apr 25, 2026 · Artificial Intelligence

How DeepSeek V4 Advances Structured Optimization in the Large‑Model Era

The article analyses DeepSeek V4’s architectural innovations—including Compressed Sparse Attention, Heavily Compressed Attention, a cross‑layer MoE design, and an Agent‑RL framework with Generative Reward Models and multi‑teacher distillation—while comparing its long‑context capabilities and efficiency to rival LLMs such as GLM, Kimi, Claude, GPT and Gemini.

Agent Reinforcement LearningCompressed Sparse AttentionGenerative Reward Model
0 likes · 7 min read
How DeepSeek V4 Advances Structured Optimization in the Large‑Model Era
Architect's Tech Stack
Architect's Tech Stack
Apr 25, 2026 · Artificial Intelligence

DeepSeek‑V4 Launch: 1.6 T Parameters, 1 M‑Token Context, Programming Skills Lead Open‑Source Rankings

DeepSeek released the V4 series—V4‑Pro (1.6 T total, 49 B active) and V4‑Flash (284 B total, 13 B active)—featuring three architectural upgrades, three inference modes, mixed‑precision FP4/FP8 weights, and benchmark results that place its programming ability at the top of open‑source models while supporting a million‑token context window.

AI ArchitectureDeepSeekbenchmark
0 likes · 5 min read
DeepSeek‑V4 Launch: 1.6 T Parameters, 1 M‑Token Context, Programming Skills Lead Open‑Source Rankings
ArcThink
ArcThink
Apr 25, 2026 · Artificial Intelligence

DeepSeek V4’s Silent Launch: 1.6 T Parameters, Triple Innovation, and Redefined Accessibility

DeepSeek V4 quietly debuted with a 1.6‑trillion‑parameter MoE model, introducing CSA+HCA compressed attention, mHC manifold‑constrained hyperconnections, and the Muon optimizer, achieving 1M‑token context at a quarter of V3’s cost, top Codeforces and LiveCodeBench scores, a 1/7 Opus price, MIT open‑source licensing, and dual‑stack Ascend NPU/NVIDIA GPU support.

Manifold-constrained HyperconnectionMuon optimizerbenchmark
0 likes · 17 min read
DeepSeek V4’s Silent Launch: 1.6 T Parameters, Triple Innovation, and Redefined Accessibility
AI Agent Super App
AI Agent Super App
Apr 24, 2026 · Artificial Intelligence

DeepSeek V4 Launches with 1.6 T Parameters and 1 Million‑Token Context

DeepSeek V4, released on April 24 2026, offers two SKUs—Pro with 1.6 T total parameters and Flash with 284 B—both supporting a 1‑million‑token context window, ultra‑low inference cost, pricing as low as ¥0.2 per million tokens, Huawei Ascend deployment, and seamless OpenAI/Anthropic API compatibility.

AI pricingAPI compatibilityDeepSeek
0 likes · 7 min read
DeepSeek V4 Launches with 1.6 T Parameters and 1 Million‑Token Context
Alibaba International Intelligent Technology
Alibaba International Intelligent Technology
Apr 24, 2026 · Artificial Intelligence

Alibaba International AI Team Lands Multiple Papers at SIGIR, WWW, and WSDM 2026

Alibaba International Intelligent Technology showcases nine industrial‑grade recommendation and search papers accepted at SIGIR, WWW, and WSDM 2026, detailing sparse scaling, counterfactual multi‑task learning, generative recommendation, MoE routing, and multimodal semantic ID breakthroughs with extensive offline and online results.

counterfactual learninge‑commerce AIgenerative recommendation
0 likes · 24 min read
Alibaba International AI Team Lands Multiple Papers at SIGIR, WWW, and WSDM 2026
SuanNi
SuanNi
Apr 21, 2026 · Artificial Intelligence

How Qwen3.6‑35B‑A3B Matches Dense Models with Only 30 B Active Parameters

The article analyzes Qwen3.6‑35B‑A3B’s MoE architecture, showing how its 30 B active parameters outperform larger dense models across programming, agent, and multimodal benchmarks, and examines the flagship Qwen3.6‑Max‑Preview’s substantial gains in world knowledge, instruction following, and third‑party rankings.

AI-evaluationQwenbenchmark
0 likes · 5 min read
How Qwen3.6‑35B‑A3B Matches Dense Models with Only 30 B Active Parameters
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Apr 21, 2026 · Artificial Intelligence

How a 22‑Year‑Old Reversed‑Engineered Mythos into OpenMythos Using MoE and DeepSeek‑Inspired Attention

OpenMythos re‑creates the Claude Mythos architecture as a Recurrent‑Depth Transformer with MoE routing, achieving comparable performance to larger Transformers while using roughly half the parameters, and demonstrates systematic generalization and depth extrapolation through looped inference in latent space.

AI ArchitectureLooped Language ModelsOpenMythos
0 likes · 6 min read
How a 22‑Year‑Old Reversed‑Engineered Mythos into OpenMythos Using MoE and DeepSeek‑Inspired Attention
AI Large-Model Wave and Transformation Guide
AI Large-Model Wave and Transformation Guide
Apr 18, 2026 · Artificial Intelligence

Does Qwen3.6‑35B‑A3B Really Outclass All AI Coding Models? Inside the Benchmark Breakdown

Qwen3.6‑35B‑A3B, a mixture‑of‑experts model that activates only 3 B parameters, outperforms leading AI systems across SWE‑bench, Terminal‑Bench, NL2Repo and several agentic coding benchmarks, while also achieving top scores in GPQA, HMMT and RealWorldQA, prompting a reassessment of domestic LLM capabilities.

AI codingAgentic CodingChinese AI
0 likes · 7 min read
Does Qwen3.6‑35B‑A3B Really Outclass All AI Coding Models? Inside the Benchmark Breakdown
Machine Heart
Machine Heart
Apr 17, 2026 · Artificial Intelligence

DeepSeek Introduces Mega MoE and FP4 Indexer – Inside the New GPU Fusion Kernel

DeepSeek's latest DeepGEMM update adds Mega MoE, a fused GPU kernel that collapses the entire Mixture‑of‑Experts pipeline and overlaps computation with NVLink communication, while also unveiling an FP4 indexer and FP8×FP4 precision experiments, signaling a push toward highly efficient large‑scale AI training.

DeepGEMMDeepSeekFP4 Indexer
0 likes · 5 min read
DeepSeek Introduces Mega MoE and FP4 Indexer – Inside the New GPU Fusion Kernel
Machine Heart
Machine Heart
Mar 31, 2026 · Artificial Intelligence

ProMoE: Explicit Routing Breaks the Scaling Bottleneck of Diffusion‑Transformer MoE (ICLR 2026)

ProMoE introduces a two‑step routing MoE framework with explicit semantic guidance that tackles the high spatial redundancy and functional heterogeneity of visual tokens, enabling diffusion transformers to scale efficiently and outperform dense models and prior MoE approaches across generation, convergence, and scaling benchmarks.

Explicit RoutingPrototypical RoutingRouting Contrastive Learning
0 likes · 9 min read
ProMoE: Explicit Routing Breaks the Scaling Bottleneck of Diffusion‑Transformer MoE (ICLR 2026)
AIWalker
AIWalker
Mar 23, 2026 · Artificial Intelligence

Dynamic Dense Computing and Minimal End‑to‑End Design: YOLO-Master & YOLO26

By introducing a dynamic mixture‑of‑experts routing scheme and an end‑to‑end architecture that eliminates NMS and DFL, YOLO‑Master and YOLO26 dramatically cut compute waste and latency on edge devices, achieving up to 43% faster CPU inference while keeping model accuracy, with all code openly released.

Computer VisionYOLOdynamic routing
0 likes · 7 min read
Dynamic Dense Computing and Minimal End‑to‑End Design: YOLO-Master & YOLO26
AIWalker
AIWalker
Mar 7, 2026 · Artificial Intelligence

YOLO-Master v2026.02 Unveils Four Innovations for SOTA Object Detection

Tencent’s YOLO-Master v2026.02 adds a Mixture‑of‑Experts architecture, zero‑overhead LoRA fine‑tuning, Sparse SAHI inference for large images, and Cluster‑Weighted NMS, delivering 3‑5× faster inference, up to 70% reduced training resources, and markedly higher detection accuracy across diverse benchmarks.

Computer VisionLoRASparse Inference
0 likes · 15 min read
YOLO-Master v2026.02 Unveils Four Innovations for SOTA Object Detection
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Feb 23, 2026 · Artificial Intelligence

System Engineering Behind Billions of Parameters: Insider Training Details from Seven Top AI Labs

This article systematically dissects the engineering decisions behind frontier large‑language‑model training—covering architecture choices, attention variants, optimizer evolution, data‑curation strategies, scaling‑law insights, and post‑training SFT/RL pipelines—based on open‑source reports from seven leading AI laboratories.

Model TrainingOptimizerlarge language models
0 likes · 26 min read
System Engineering Behind Billions of Parameters: Insider Training Details from Seven Top AI Labs
AI Engineering
AI Engineering
Feb 12, 2026 · Artificial Intelligence

MiniMax M2.5: 230B‑Parameter Model Activates 10B, Near Claude Sonnet for One‑Tenth the Cost

MiniMax’s new open‑source M2.5 model, built on a 230 billion‑parameter mixture‑of‑experts architecture that activates only 10 billion parameters per inference, delivers performance comparable to Claude Opus 4.6 across benchmarks, while costing roughly one‑tenth as much, and is already handling a large share of the company’s internal tasks.

AI agentsClaude OpusMiniMax M2.5
0 likes · 6 min read
MiniMax M2.5: 230B‑Parameter Model Activates 10B, Near Claude Sonnet for One‑Tenth the Cost