Skip to content
OptiMer: Merging Distribution Vectors Beats Data Mixing for Continual Pre-Training

OptiMer: Merging Distribution Vectors Beats Data Mixing for Continual Pre-Training

NICT researchers propose OptiMer, a method that merges distribution vectors from separately trained models to find optimal data mixing ratios for continual pre-training, achieving 15-35x speedup over traditional data mixing and outperforming it on benchmarks like C-Eval, MMLU, and GSM8K.

Machine Heart
Machine Heart
Machine Heart

Introduction

Choosing the right proportion of each dataset in continual pre-training is critical but costly: ratios must be set before training, yet their quality is only known after training finishes. If results are unsatisfactory, practitioners must adjust ratios and retrain from scratch, incurring huge trial-and-error costs.

The paper "OptiMer: Optimal Distribution Vector Merging Is Better than Data Mixing for Continual Pre-Training" by Haiyue Song, Huy Hien Vu, and Masao Utiyama from Japan's National Institute of Information and Communications Technology (NICT) addresses this problem. It won the EMNLP 2026 Outstanding Paper Award, one of only 22 papers selected from 17,661 submissions.

Method: From Merging Datasets to Merging Distribution Vectors

OptiMer operates in three steps:

Train models separately on each dataset. Starting from a base model, continual pre-training is run independently on each dataset (e.g., Japanese, Chinese, math), yielding one model per dataset.

Extract distribution vectors. For each trained model, subtract the base model's parameters to obtain a distribution vector that captures the capability change induced by that dataset.

Automatically search for optimal merging weights. A new model is formed by adding weighted distribution vectors (plus an instruction-following vector) to the base model. Bayesian optimization adjusts the weights to maximize validation performance. Each trial only requires model merging and evaluation — no retraining.

OptiMer method overview
OptiMer method overview

Efficiency: 15–35× Speedup

After initial per-dataset training, traditional data mixing needs 129.3 hours to train a single ratio. OptiMer tests 100 ratios in just 8.5 hours, delivering a 15–35× acceleration in the ratio-optimization phase.

Efficiency comparison chart
Efficiency comparison chart

Performance: Better Than Data Mixing

On standard benchmarks (C-Eval, Japanese Leaderboard, MMLU, GSM8K, HumanEval), OptiMer achieves an average score of 70.37 versus 63.71 for traditional data mixing — a 6.7-point gain.

Moreover, the optimal merging weights found by OptiMer can guide data mixing ratios directly. Using those weights as mixing proportions and retraining lifts the average score from 63.71 (equal-proportion mixing) to 68.66.

Performance comparison chart
Performance comparison chart

Flexibility: Rapid Re-optimization for New Tasks

Traditional data mixing requires full retraining for each new task. OptiMer reuses the already-trained distribution vectors; only the evaluation objective changes, and Bayesian optimization adjusts the merging weights. For example, optimizing for English tasks yields the best English performance; optimizing for math yields strong math performance; optimizing for Japanese quickly produces a Japanese-specialized model. This is highly valuable in industrial deployment where multiple domain-specific models (education, finance, etc.) are needed.

Analysis 1: Why Distribution Vectors Can Be Merged

Cosine similarity between distribution vectors and the instruction-following vector is only ~0.03 (near orthogonal); between different distribution vectors it is ~0.3. The authors hypothesize that existing merging algorithms like DARE can combine these vectors with minimal interference.

Cosine similarity analysis
Cosine similarity analysis

Analysis 2: Principal Component Analysis of Distribution Vectors

PCA projects high-dimensional vectors onto a 2D plane. Distribution vectors from different datasets and the instruction-following vector lie far apart. OptiMer's merging weights assign a large weight to the instruction-following vector and smaller weights to data distribution vectors, placing the optimal merged model near the line between the base model and the instruction-tuned model — suggesting high-quality models reside in that region.

PCA visualization
PCA visualization

Analysis 3: OptiMer's Search Process

Bayesian optimization concentrates search in a narrow high-performing region (large instruction-following weight, small data-distribution weights). It requires roughly 10×n evaluations (n = number of datasets), far fewer than grid search, which easily misses the optimum especially with many datasets. The search runs in parallel batches of 8 points: early exploration, rapid mid-phase convergence, and final exploitation to find the best score.

Search process visualization
Search process visualization

Analysis 4: Vector Changes During Training

Tracking models at different training steps shows that continual pre-training trajectories are approximately straight lines in PCA space. Scaling a distribution vector's weight mimics increasing or decreasing effective training on that dataset. Vector norm grows with training steps, while model performance peaks early then declines, suggesting models should not stray too far from the base model — consistent with Analysis 2.

Training trajectory analysis
Training trajectory analysis

Analysis 5: Extending to Negative Weights

Allowing negative weights sometimes improves results further, hinting at machine unlearning — selectively forgetting certain knowledge to benefit the target task. The paper does not explore this deeply but flags it as an intriguing direction.

Negative weight exploration
Negative weight exploration

Application: Building a Japanese-Specialized Model

OptiMer was applied to the Gemma-SEA-LION-v4-27B model (a Gemma 3 variant continually pre-trained on Southeast Asian languages by AI Singapore). The goal was to enhance Japanese ability while preserving Southeast Asian language performance. Distribution vectors were trained on Japanese, Chinese, English, math, and code data, then merged with the SEA-LION instruction-following vector. Bayesian optimization targeting Japanese tasks raised the Japanese score from 66.34 to 74.40, with Southeast Asian language performance retained.

Japanese specialization results
Japanese specialization results

Conclusion

OptiMer adapts open-source LLMs to target languages or domains more efficiently, effectively, and flexibly than traditional data mixing. Experiments are limited to continual pre-training; future work aims to extend the approach to post-training to solve optimal ratio problems for the massive datasets used there.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

large language modelsBayesian optimizationdata mixingmodel mergingEMNLP 2026continual pre-trainingdistribution vectorsOptiMer
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.