Train a 64M LLM from Scratch in 2 Hours for $3 and Master LLM Systems

MiniMind: Train a Large Model in 2 Hours for $3

MiniMind is an ultra‑lightweight LLM reproduction project that enables anyone to train a usable language model from scratch. The smallest version is only 1/2700 the size of GPT‑3, allowing training on a single consumer GPU.

Key Characteristics

Training cost under $3 (GPU rental for one epoch on a 3090)

Training time roughly 2 hours

All core code is handwritten, with no reliance on high‑level wrappers

Complete Training Pipeline

Pre‑training, Supervised Fine‑Tuning (SFT), LoRA, RLHF (DPO), RLAIF (PPO/GRPO/CISPO)

Supports tool use and agent‑based reinforcement learning

Supports model distillation and adaptive reasoning

Supported Architectures

Dense and Mixture‑of‑Experts (MoE) models

Alignment with Qwen3/Qwen3‑MoE ecosystem

Parameter scales from 26 M to 198 M

Published Models

minimind‑3 – 64 M parameters

minimind‑3‑moe – 198 M (A64 M)

minimind2‑small – 26 M

minimind2‑moe – 145 M

minimind2 – 104 M

Training Results

The project provides full training logs showing loss curves for both pre‑training and SFT stages.

Performance Evaluation

Benchmarks on C‑Eval, C‑MMLU, OpenBookQA and other suites are visualized in a radar chart.

Agent Capability Demo

MiniMind includes a Streamlit‑based chat WebUI that shows reasoning steps, tool selection, and multi‑turn tool calls.

Training Data

The repository releases a high‑quality dataset with collection, distillation, cleaning, and deduplication pipelines.

Getting Started

# Clone the project
git clone https://github.com/jingyaogong/minimind
cd minimind
# Download model
./download_model.sh
# Train (2 h on a single GPU)
python pretrain.py
# Inference
python inference.py

Happy‑LLM: A Systematic LLM Tutorial

Happy‑LLM, produced by Datawhale, focuses on understanding LLM theory and practice. It has attracted about 30 K stars.

Project link: https://github.com/datawhalechina/happy-llm

What You Will Learn

Transformer architecture and attention mechanisms

Fundamentals of pre‑training language models

Comparative analysis of existing large models

Implementing a full LLaMA‑2 model from scratch

End‑to‑end workflow from pre‑training to fine‑tuning

Frontier applications such as RAG and agents

Content Navigation

Chapter 1 – NLP basics

Chapter 2 – Transformer architecture

Chapter 3 – Pre‑training language models

Chapter 4 – Large language models

Chapter 5 – Hands‑on model building

Chapter 6 – LLM training practice

Chapter 7 – LLM applications

Key Features

Completely free : Datawhale open‑source project

Theory + Practice : Both conceptual explanations and code implementations

Supplementary blog : Extra Chapter collects excellent notes and blogs

Continuous updates : Active community adds new content regularly

Relationship Between the Two Projects

Happy‑LLM – focuses on theoretical learning and code practice; suited for developers who want a systematic understanding of LLM principles.

MiniMind – provides a full training workflow and low‑cost reproducibility; suited for developers who want to train a model hands‑on.

Combined, they cover the entire spectrum from theory to implementation.

Both projects are valuable resources for anyone learning large‑model fundamentals or wishing to train LLMs from scratch.