OptiMer: Merging Distribution Vectors Beats Data Mixing for Continual Pre-Training
NICT researchers propose OptiMer, a method that merges distribution vectors from separately trained models to find optimal data mixing ratios for continual pre-training, achieving 15-35x speedup over traditional data mixing and outperforming it on benchmarks like C-Eval, MMLU, and GSM8K.
