Hybrid Attention: Why Kimi and DeepSeek Now Share a Model Architecture

This article traces the evolution of attention mechanisms in large language models, showing how hybrid architectures now combine linear and sparse attention — exemplified by GLM-5.3-Flash integrating Kimi's KDA and DeepSeek's DSA — driven by shifting constraints from context length to agent workloads, with MiniMax's architectural journey illustrating the trade-offs.

Context ScalingDeepSeekHybrid Architecture

0 likes · 17 min read

Hybrid Attention: Why Kimi and DeepSeek Now Share a Model Architecture

AI2ML AI to Machine Learning

Nov 3, 2025 · Artificial Intelligence

Smol Training Playbook: Secrets to Building World-Class LLMs

The article details the SmolLM3 3B‑parameter model, its architecture, dual‑mode inference, a three‑stage data‑curation strategy, rigorous ablation methods, preference optimisation (APO/DPO), model merging, and practical training‑stability tricks, offering a comprehensive guide for building high‑performing large language models.

APOContext ScalingLLM Training

0 likes · 13 min read

Smol Training Playbook: Secrets to Building World-Class LLMs