Baobao Algorithm Notes
May 21, 2024 · Artificial Intelligence
How to Pre‑train a 20M‑Parameter LLaMA‑3 Mini Model with Hugging Face Trainer
This step‑by‑step guide shows how to use Hugging Face's Trainer API to pre‑train an ultra‑small LLaMA‑3 model (under 20 M parameters) on the TinyStories dataset, covering model configuration, tokenizer setup, data preprocessing, collators, training arguments, and inference results.
Hugging FaceLLaMALanguage Model
0 likes · 27 min read