BestHub
Sign in
Tagged articles

MLA Attention

3 articles · Page 1 of 1
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Aug 5, 2026 · Artificial Intelligence

Inside K3: How Stable Latent MoE and MLA Attention Are Designed

The article examines K3’s architecture—combining KDA, MLA, Stable Latent MoE and AttnRes—detailing the replacement of SwiGLU with SiTU‑GLU, the addition of RMSNorm for training stability, the Quantile Balancing load‑balancing scheme, and the trade‑offs behind its MLA and NoPE attention choices.

K3MLA AttentionMixture of Experts
0 likes · 15 min read
Inside K3: How Stable Latent MoE and MLA Attention Are Designed
Geek Labs
Geek Labs
Jul 14, 2026 · Artificial Intelligence

How Colibri Runs a 744 B MoE Model on CPU with Only 25 GB RAM

Colibri demonstrates that the 744‑billion‑parameter GLM‑5.2 MoE model can be executed on a pure‑CPU system using roughly 25 GB of memory by streaming expert weights from disk, int4‑quantizing dense parameters, and employing a compact C inference engine.

CPU inferenceGLM-5.2MLA Attention
0 likes · 7 min read
How Colibri Runs a 744 B MoE Model on CPU with Only 25 GB RAM
Alibaba Cloud Developer
Alibaba Cloud Developer
Feb 7, 2025 · Artificial Intelligence

Why DeepSeek V3 Achieves Low Training Costs: Inside Its AI Innovations

This article provides a comprehensive analysis of DeepSeek's large‑language‑model technology, covering the company's background, model capabilities, remarkably low training and inference costs, and the core architectural and algorithmic innovations such as MoE, MLA attention, FP8 mixed‑precision, and the DualPipe pipeline that enable efficient large‑scale AI deployment.

AI architectureDeepSeekFP8 training
0 likes · 19 min read
Why DeepSeek V3 Achieves Low Training Costs: Inside Its AI Innovations