NewBeeNLP

Jul 31, 2024 · Artificial Intelligence

How Continual Pre‑Training Boosts Llama‑3’s Chinese and Scientific Reasoning

This report presents a continual pre‑training approach that significantly enhances Llama‑3 (8B)’s Chinese language proficiency and scientific reasoning by using a carefully mixed corpus of existing and synthetic data, detailing the bilingual adaptation and synthetic‑enhancement stages, data‑mixing and curriculum strategies, and demonstrating strong results across multilingual and scientific benchmarks without sacrificing original capabilities.