Training 7B–13B LLMs: Practical Tips, Hyperparameters, and Scaling Challenges

Since the emergence of BERT models with hundreds of millions of parameters, the field has rapidly moved to ever larger models such as GPT‑3 (170 B parameters) and a new wave of competitors: Google’s trillion‑parameter sparse Switch Transformer, Huawei’s 200 B dense PanGu‑α, Microsoft’s Turing‑NLG (1 T parameters), and NVIDIA’s Megatron‑LM series.

Drawing from personal experience training models of 7 B and 13 B parameters, the author notes that the training process is similar to smaller models but requires two key hyper‑parameter adjustments: a slightly lower learning rate (around 1e‑5) and a large global batch size (2–4 M tokens) to maintain stability.

For a 7 B model the typical configuration is stage = 2, tensor‑parallelism = 1, ZeRO‑1 = 8. This fits a single 80 GB GPU with the full model parameters (≈14 GB) and distributes optimizer states across eight GPUs, resulting in roughly 13 GB per GPU. With a sequence length of 2048, each GPU can handle a micro‑batch size of about 8.

If gradient accumulation is omitted, achieving a global batch of 2 M tokens would require at least 128 GPUs (8 × 2048 × 128). In practice, such a setup can process 30–40 B tokens in 24 hours, meaning a trillion tokens would take roughly a month, assuming no hardware failures.

Scaling beyond the billion‑parameter range introduces additional challenges: numerical instability, convergence issues, and a dramatically higher fault rate as the number of machines grows. Reports such as the GLM‑130B training log and Meta’s OPT logbook illustrate the frequent failures encountered when coordinating thousands of GPUs.

Beyond compute, data quality becomes the dominant factor for model performance at a given scale. While many English corpora (C4, The Pile, RefinedWeb) are openly available, high‑quality Chinese data is scarcer. Open Chinese datasets mentioned include:

WuDaoCorpus (≈200 GB Chinese text) [1] https://data.baai.ac.cn/details/WuDaoCorporaText

TigerBot (≈100 GB mixed Chinese‑English) [2] https://github.com/TigerResearch/TigerBot?tab=readme-ov-file#%E9%A2%84%E8%AE%AD%E7%BB%83%E6%95%B0%E6%8D%AE

SkyPile‑150B (150 B tokens) [3] https://huggingface.co/datasets/Skywork/SkyPile-150B

WanJuan (≈1 TB Chinese text, currently the largest open Chinese corpus) [4] https://github.com/opendatalab/WanJuan1.0/blob/main/WanJuan1.0-CN.md

CCI (104 GB Chinese text) [5] https://data.baai.ac.cn/details/BAAI-CCI

However, Chinese data still lags behind English in both volume and cleanliness; noisy content such as advertisements, AIGC‑generated text, factual errors, and unsafe material is prevalent, necessitating dedicated data‑curation teams.

The trial‑and‑error cost of training large models is extremely high. Even with substantial compute, time, and personnel invested, there is no guarantee of superior model performance because data quality is hard to quantify. Practitioners must monitor loss curves continuously and be prepared to augment training data iteratively.

Looking ahead, open‑source foundation models will likely need to start from at least 2 T tokens to remain competitive according to scaling laws, reducing the practical impact of smaller‑scale releases. Ultimately, the difficulty of training massive models is not a single technical hurdle but a systemic one that demands tight coordination among data engineers, model developers, framework experts, and hardware resources. This systemic nature explains why startups often achieve breakthroughs faster than large enterprises, which can suffer from internal friction.