Skip to content
Tagged articles

data mixing

4 articles · Page 1 of 1
Machine Heart
Machine Heart
Nov 14, 2026 · Artificial Intelligence

OptiMer: Merging Distribution Vectors Beats Data Mixing for Continual Pre-Training

NICT researchers propose OptiMer, a method that merges distribution vectors from separately trained models to find optimal data mixing ratios for continual pre-training, achieving 15-35x speedup over traditional data mixing and outperforming it on benchmarks like C-Eval, MMLU, and GSM8K.

Bayesian optimizationEMNLP 2026OptiMer
0 likes · 12 min read
OptiMer: Merging Distribution Vectors Beats Data Mixing for Continual Pre-Training
PaperAgent
PaperAgent
Nov 1, 2026 · Artificial Intelligence

CMU's Unified LLM Training: One Maximum Likelihood Objective for All Stages

CMU researchers propose a unified training paradigm where pre-training, mid-training, supervised fine-tuning, and reinforcement learning share a single maximum likelihood objective, using data mixing and scheduling instead of stage-specific algorithms, and show improved reasoning and continual learning on puzzles, math, and code benchmarks.

CMUContinual LearningMaxRL
0 likes · 9 min read
CMU's Unified LLM Training: One Maximum Likelihood Objective for All Stages
DataFunSummit
DataFunSummit
Feb 5, 2025 · Artificial Intelligence

Exploration and Practice of Large‑Model Data Construction

This presentation details engineering‑focused approaches to building, mixing, and filtering data for large language models, covering data preparation, pre‑training mix strategies such as DoReMi, DoGE and online sampling, post‑training data quality selection methods, and practical Q&A on scaling laws and PDF processing.

AIData EngineeringModel Scaling
0 likes · 15 min read
Exploration and Practice of Large‑Model Data Construction
Baobao Algorithm Notes
Baobao Algorithm Notes
Mar 29, 2024 · Artificial Intelligence

Can Data Mixing Laws Predict LLM Performance? A Deep Dive into Scaling Laws

This article reviews the paper “Data Mixing Laws: Optimizing Data Mixture by Predicting Language Modeling Performance”, explaining how the authors quantify the impact of data mixture ratios on LLM loss, propose a simple predictive model, validate it on RedPajama and multi‑domain mixes, and outline a scaling‑law procedure for continual pre‑training.

Data SchedulingLLMdata mixing
0 likes · 9 min read
Can Data Mixing Laws Predict LLM Performance? A Deep Dive into Scaling Laws