523 Lessons: Build a Working GPT from Scratch - Free AI Engineering Course

Course Overview

The free Chinese course "Learn AI Engineering from Scratch" (learn.wink.run) adapts the open-source GitHub course "AI Engineering from Scratch" by Rohit Ghumare (MIT License). It comprises 523 lessons across 20 stages, requiring no registration; progress is stored locally in the browser. The course philosophy is "write every line yourself," emphasizing hands-on implementation over passive reading.

Curriculum Structure

The first 19 stages cover major AI engineering domains:

Environment and toolchain (12 lessons)

Mathematical foundations (22 lessons)

Machine learning fundamentals (18 lessons)

Deep learning core (13 lessons)

Computer vision (28 lessons)

NLP (29 lessons)

Speech and audio (17 lessons)

Transformer deep dive (16 lessons)

Generative AI (15 lessons)

Reinforcement learning (12 lessons)

Stages 11–18 focus heavily on LLMs and agents:

Hand-written LLM (24 lessons)

LLM application engineering (17 lessons)

Multimodal (25 lessons)

Tools and protocols (31 lessons)

Agent engineering (54 lessons)

Autonomous systems (22 lessons)

Multi-agent and swarm (25 lessons)

Infrastructure and production (28 lessons)

Ethics, safety, and alignment (30 lessons)

The final stage (stage 19) contains 85 capstone projects. The first 17 are large projects estimated at 25–40 hours each (e.g., terminal-native coding agent, real-time voice assistant, Kubernetes troubleshooting agent, speculative decoding inference server). The remaining 68 lessons are ~90 minutes each, grouped to build complete systems.

Key Features

Searchable curriculum: The table of contents supports keyword search (e.g., "RAG" highlights 10 lessons across multiple stages).

Reading aids: Long lessons include a sticky table of contents that tracks scroll position; code blocks have one-click copy buttons.

Interactive diagrams: Animations illustrate processes like ring allreduce (reduce-scatter and allgather steps) and Transformer block residual flows (pre-LN structure with two residual branches).

Instant quizzes: Multiple-choice questions are graded locally with immediate feedback (checkmarks for correct, crosses for incorrect); answers are not uploaded.

Dark mode and mobile support: Every page offers a dark theme; mobile layout uses a single column with a bottom toolbar for navigation.

Capstone Project Lines

Stage 19 includes several end-to-end project sequences:

Train a GPT from scratch: BPE tokenizer, sliding-window dataset, token and positional embeddings, multi-head self-attention, Transformer blocks assembled into a 124M-parameter GPT (vocab 50257, context 1024, embedding 768, 12 heads, 12 layers), then load pretrained weights for classification fine-tuning, SFT, and DPO.

Training infrastructure: Large corpus download, HDF5 token storage, cosine learning rate with warmup, gradient clipping and mixed precision, gradient accumulation, checkpoint save and restore.

Coding agent: Agent harness loop, schema-validated tool registry, JSON-RPC over stdio, sandboxing, evaluation, and OpenTelemetry observability, culminating in an end-to-end coding agent.

Autonomous research agent: Hypothesis generation, literature retrieval, experiment execution, result evaluation, paper writing, and critic loop.

Multimodal: Image patching, ViT encoder, modality alignment projection, cross-attention fusion, vision-language pretraining and evaluation.

Production-grade RAG: Chunking strategy comparison, hybrid retrieval (BM25 + vector), cross-encoder reranking, HyDE query rewriting, evaluated with precision, recall, MRR, nDCG, and faithfulness.

Evaluation system: Task formats, classic metrics, code execution scoring, perplexity and calibration, leaderboard aggregation.

Distributed training: Implement collective communication from scratch, then DDP, ZeRO optimizer state sharding, pipeline parallelism with bubble analysis, sharded checkpoints, and a full end-to-end distributed training run.

AI safety gate: Jailbreak classification, prompt injection detection, refusal evaluation, content classifier, constitutional rule engine, combined into a three-checkpoint gate (pre-generation, during-generation, post-generation).

Lesson Example: GPT Model Assembly

The lesson "GPT Model Assembly" shows the typical layout: left sidebar with stage lessons, center content, right sidebar with page table of contents and stage progress. The lesson header lists type, language, prerequisites, and duration. Learning objectives include reproducing a 124M-parameter configuration (vocab 50257, context 1024, embedding 768, 12 heads, 12 layers). Content follows a problem–concept–implementation–usage–exercise flow. Code blocks are annotated with language and have a copy button. Interactive diagrams animate collective communication and Transformer block internals.

Target Audience and Learning Paths

The course suits developers who want to build AI engineering systems hands-on. Recommended entry paths:

Foundation camp: Stages 0–3 (65 lessons) – environment setup, math review, classic ML, hand-written neural networks.

Hand-written to Transformer: Stages 5 and 7 (45 lessons).

LLM application engineering: Stage 11 (17 lessons) – prompting, RAG, evaluation, guardrails, cost.

Agent and MCP: Stages 13–14 (85 lessons).

Learners with existing background can jump directly to stage 19 and pick a capstone project.

Source Attribution

Content is adapted from Rohit Ghumare's open-source "AI Engineering from Scratch" (MIT License) at github.com/rohitg00/ai-engineering-from-scratch. This site is an independent Chinese companion, not an official translation.