Skip to content
BestHub
Sign in
Tagged articles

LLM Pretraining

7 articles · Page 1 of 1
SmartApe AI
SmartApe AI
Nov 13, 2026 · Artificial Intelligence

No Second Run: RSI for LLM Pretraining Yields 30B MoE Model in 3 Months

Shanghai AI Lab and Xinxu AI introduce No Second Run, a recursive self-improvement method for LLM pretraining using debugging, headroom evaluation, and experience-to-metric loops. In three months they trained the 801-30B-A3B MoE model (30B total, 3B active params) on 11.8T tokens, scoring 73.13 average on 13 base benchmarks (second only to Qwen3.5-35B-A3B) and achieving top Agent benchmark results after 6.33B-token SFT without RL.

801-30B-A3BAgent BenchmarksLLM Pretraining
0 likes · 15 min read
No Second Run: RSI for LLM Pretraining Yields 30B MoE Model in 3 Months
Data Party THU
Data Party THU
May 29, 2026 · Artificial Intelligence

Token Superposition Training: 2.5× Faster LLM Pre‑training Without Model Changes

The article presents Token Superposition Training (TST), which temporarily averages embeddings of non‑overlapping token bags and predicts groups of tokens in a first phase before reverting to standard token‑wise prediction, achieving up to 2.5× pre‑training speedup on 10B‑1B MoE models without altering model architecture or inference.

LLM PretrainingMCE lossMixture of Experts
0 likes · 9 min read
Token Superposition Training: 2.5× Faster LLM Pre‑training Without Model Changes
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
May 16, 2026 · Artificial Intelligence

Token Superposition Training Accelerates LLM Pre‑training 2.5× Without Changing Architecture

Token Superposition Training (TST) speeds up large‑language‑model pre‑training by up to 2.5× without altering model architecture or compute budget, using a superposition phase that averages token embeddings into bags and predicts groups of tokens, followed by a standard recovery phase, as demonstrated on 10B‑parameter MoE and smaller models.

LLM PretrainingMCE lossMoE
0 likes · 10 min read
Token Superposition Training Accelerates LLM Pre‑training 2.5× Without Changing Architecture
Baobao Algorithm Notes
Baobao Algorithm Notes
Dec 23, 2024 · Artificial Intelligence

From Zero to One: A Practical Guide to Pretraining Large Language Models

This comprehensive guide walks through every stage of building a large‑language‑model pretraining pipeline—from data sourcing, cleaning, and deduplication, to tokenizer design, model architecture choices, training framework selection, optimization tricks, and evaluation methods—providing actionable tips and pitfalls to avoid for both newcomers and seasoned practitioners.

LLM Pretrainingdata collectionscaling laws
0 likes · 33 min read
From Zero to One: A Practical Guide to Pretraining Large Language Models
NewBeeNLP
NewBeeNLP
Sep 25, 2024 · Artificial Intelligence

From Zero to One: A Practical Guide to Pretraining Large Language Models

This comprehensive guide walks through every stage of LLM pretraining—from data sourcing, cleaning, and deduplication, to tokenizer design, model architecture choices, training framework selection, optimization tricks, and evaluation methods—offering actionable tips and pitfalls to avoid.

LLM Pretrainingdata collectionevaluation methods
0 likes · 32 min read
From Zero to One: A Practical Guide to Pretraining Large Language Models
Baobao Algorithm Notes
Baobao Algorithm Notes
Sep 24, 2024 · Artificial Intelligence

From Zero to One: A Practical Guide to Pretraining Large Language Models

This comprehensive guide walks you through every stage of LLM pretraining—from data sourcing, cleaning, and deduplication to tokenizer design, model architecture choices, training framework selection, optimization tricks, and evaluation methods—highlighting common pitfalls and practical solutions for building robust models.

LLM Pretrainingcurriculum learningdata cleaning
0 likes · 34 min read
From Zero to One: A Practical Guide to Pretraining Large Language Models