Why Llama 3’s Open‑Source Release Could Redefine Large‑Model Scaling and Synthetic Data

Llama 3 Overview

Llama 3 retains the same core architecture as Llama 2 but introduces several key changes: the token vocabulary expands from 32K to 128K to improve encoding efficiency, Grouped Query Attention (GQA) reduces KV‑cache size and speeds up inference, and the maximum context length doubles from 4K to 8K tokens. The most significant upgrade is the training data volume, which grows from 2 T tokens in Llama 2 to roughly 15 T tokens—about eight times larger, with code data increasing fourfold—resulting in markedly better coding and logical reasoning abilities. Llama 3 is released in three sizes: an 8 B parameter model that slightly outperforms Mistral 7B/Gemma 7B, a 70 B model whose performance sits between ChatGPT 3.5 and GPT‑4, and a 400 B model (still in training) aimed at multimodal, multilingual capabilities comparable to GPT‑4/4V.

Open‑Source vs Closed‑Source

Meta positions itself as a pillar of the open‑source large‑model community and plans to open‑source the entire Llama 3 family, including the 400 B model, within months. This promises an open‑source model with performance close to GPT‑4, offering a strong alternative for complex applications. If Meta continues this openness for future generations (e.g., Llama 4), Chinese researchers should focus on better localizing Llama—expanding the Chinese token set, continuing pre‑training with low‑cost Chinese data, and removing harmful content—to create Chinese‑adapted models that could surpass many domestic closed‑source offerings. While open‑source models currently lag behind closed‑source ones, the performance gap has been narrowing over the past year and a half. The decisive factor is the “model‑ability acceleration”: a steep improvement curve favors closed‑source models with massive compute resources, whereas a flatter curve allows open‑source models to catch up more quickly.

Synthetic Data and Future Scaling

Synthetic data is an emerging, still‑immature research direction. Existing high‑impact examples include DALLE‑3 and Sora, where generated images or videos serve as training data. Investing heavily in synthetic‑data techniques is seen as a hedge against a potential shortage of high‑quality new data after 2025. If synthetic‑data breakthroughs occur within the next two years, the gap between open and closed models could widen; if not, both camps may converge, and model improvements will rely mainly on scaling model size rather than data volume, albeit with slower efficiency gains. The article outlines two future scenarios: (1) synthetic data remains impractical, leading to a plateau in model capabilities and heightened pressure on closed‑source providers; (2) synthetic data or new data‑utilization methods advance, allowing continued scaling of data and model size, keeping the path toward AGI viable but demanding astronomical compute investment.

https://www.zhihu.com/question/653373334/answer/3471466524