System Engineering Behind Billions of Parameters: Insider Training Details from Seven Top AI Labs

Architecture Foundations and Attention Mechanisms

When compute is limited or absolute stability is required, a dense backbone combined with GQA and RoPE/RNOPE is the most reliable choice. MoE architectures offer inference‑time efficiency but depend heavily on global load‑balancing and routing strategies; the granularity of expert partitioning is the key metric. GQA with small groups (2, 4, 8) consistently outperforms MHA and MQA at the same scale, while MQA saves memory at the cost of attention‑capacity leakage. MLA compresses the KV cache 4–8× with comparable performance, but at the expense of considerable implementation complexity. Gated attention applies element‑wise gates to the scaled‑dot‑product output, mitigating attention‑sink issues, especially on long sequences.

Document masking is the standard remedy for cross‑document attention leakage; it yields modest gains on short‑context tasks but becomes critical when the context window expands from 4 k to 64 k tokens. Embedding sharing dramatically reshapes parameter allocation—untied embeddings can occupy up to 20 % of parameters in smaller models. Experiments on Hugging Face’s 1.2 B model show that tying embeddings reduces parameters by 18 % while preserving competitive performance, whereas the untied variant suffers higher training loss and downstream degradation.

Positional encodings for long text are handled by RoPE, which rotates query/key vectors proportionally to position. YaRN dominates ultra‑long context extensions, enabling gpt‑oss‑120b to stretch dense layers to 131 k tokens. RNOPE alternates RoPE and NoPE to retain local perception while improving long‑range retrieval.

Training Stability and Optimizer Evolution

Early z‑loss attempts to penalize large softmax denominators showed no measurable benefit at the 1 B scale. Modern pipelines have shifted to Logit Softcapping (e.g., Gemma 2) where a tanh clamp limits logits (layer‑wise thresholds: 50.0 for attention layers, 30.0 for final layers). This breaks compatibility with FlashAttention, forcing training back to eager mode.

QK‑norm, once used to suppress logits, removes magnitude information from query‑key products, harming long‑text performance. Recent work introduces Sandwich Norm (Arcee, Trinity Large) applying RMSNorm before and after attention/MLP blocks. AdamW remains the default adaptive optimizer, while Muon treats the weight matrix as a geometric object and uses Newton‑Schulz iterations (a fifth‑order polynomial) to approximate the matrix sign function, yielding faster convergence on large batches.

Load‑balancing failures cause capacity collapse; three technical routes have emerged: loss‑based balancing (matching average routing probability to actual selection ratio), bias‑only balancing (cutting off interfering gradients), and SMEBU’s soft‑clipping of the step‑size for the symbolic function, coupled with momentum buffers to smooth noise.

Data Strategies and Mid‑training

Beyond architecture, data quality ultimately determines model performance. Over‑filtering for high‑quality data can lead to excessive duplication and reduced generalisation. OpenAI’s gpt‑oss‑120b employed strict filtering to remove hazardous biochemical knowledge. Multi‑stage training injects the highest‑quality, reasoning‑focused data late in training to shape final behaviour.

SmolLM3’s 7 T‑token checkpoint used a 75/12/10/3 split (English web, multilingual web, code, math). English data mixed FineWeb‑Edu and DCLM at 60/40 or 50/50 ratios; code data (Stack‑Edu) was delayed to later stages with a 40/60 mix. Token utility measures the contribution of each token to learning signals. Kimi K2 rewrites knowledge data with diverse prompts, limits re‑generation of the same passage to two instances, and splits mathematical data into note‑style fragments with cross‑language translation to enrich representations.

Scaling laws provide the baseline FLOPs estimate C ≈ N·D, where N is parameter count and D is token count. Batch size and optimal learning rate scale together; doubling batch size requires doubling the learning rate. Early training benefits from small‑batch, high‑frequency updates, while later stages tolerate larger batches with a warm‑up schedule.

Post‑training Paradigms: SFT and Reinforcement Learning

SFT establishes baseline behaviour; SmolLM3 uses a learning rate 20× lower than pre‑training (1e‑6) to balance reasoning and non‑reasoning modes. For large vocabularies, Cute Cross‑Entropy (CCE) avoids materialising the full logit matrix, computing only the correct token’s logit. SonicMoE kernels further improve compute utilisation.

Preference optimisation (PO) and ORPO integrate advantage ratios into cross‑entropy, eliminating reference model overhead. APO‑zero boosts positive sample probability while suppressing negatives; APO‑down reduces both when low‑quality positives dominate. KTO discards pairwise comparisons, updating from single‑output expectations.

Length penalties are essential for RL‑trained models to prevent runaway token generation. MuonClip, introduced in Kimi K2, caps per‑head inputs; exceeding the hard threshold incurs a –1 penalty, nullifying reward. DeepSeek‑R1‑Zero demonstrates pure RL feasibility with a stable hyper‑parameter set (10.4 k steps, batch 512, reference replacement every 400 steps, lr 3e‑6, KL 0.001) and enforces a summary at the end of each response.

Alignment and Engineering Practices

Minor template tweaks can dramatically alter model persona; replacing the assistant token with “me” gave Hermes 4 a first‑person character and reduced disclaimer output. Prompting with over‑flattering language shifts the model’s reasoning chain toward divergent, emotive responses.

Safety evaluations (Preparedness framework) incorporate network‑attack simulations and bio‑risk data. The StrongReject dataset enforces a strict hierarchy: system > developer > user > assistant > tool. Hardware stress tests and S3 checkpoint auto‑offload safeguard storage and cluster stability.

Training throughput bottlenecks were traced to Nanotron’s unbounded lookup‑table growth and Tokenizedbytes data loaders; switching loaders restored throughput. vLLM’s shared queue limited scaling across nodes, prompting a redesign where each inference node runs an independent vLLM engine with its own KV cache, orchestrated by a round‑robin scheduler.

Conclusion

From granular architectural adjustments to large‑scale cluster engineering, breakthroughs in frontier LLMs rely on exhaustive ablation studies. Systematic load‑balancing, robust RL objectives, and disciplined data pipelines together form a solid technical foundation for scaling artificial intelligence.

Reference

[1] Alex Wa. (2026). Frontier model training methodologies. https://djdumpling.github.io/2026/01/31/frontier_training.html