A Complete Open‑Source Guide to LLM Internals: From Tokenization to Inference Optimization

Learning Path

The open‑source repository defines eleven sequential topics that together form a systematic curriculum for understanding large language model (LLM) internals.

Stage 1 – Tokenization – Explains the Byte‑Pair Encoding (BPE) algorithm with concrete numeric examples, showing how models such as GPT and Claude first split raw text into sub‑words.

Stage 2 – Attention Mechanism – Derives the mathematics of the query (Q), key (K), and value (V) matrices, then step‑by‑step shows why the softmax is scaled by √dₖ. The tutorial also walks through causal masking, including a code illustration of how autoregressive models hide future tokens.

Stage 3 – Training and Back‑Propagation – Provides a full derivation of back‑propagation from the chain rule to gradient descent, using explicit numeric values for each intermediate gradient. A complete Python implementation reproduces the derivation.

Stage 4 – Transformer Architecture – Presents a panoramic view of the encoder‑decoder design, compares three Transformer variants, and details how each component (self‑attention, feed‑forward, positional encoding, etc.) collaborates within the model.

Stage 5 – Inference Optimization – Covers three key techniques:

KV Cache – Describes how keys and values are cached during generation so the model can “remember” previous tokens.

Paged Attention – Explains vLLM’s core solution for KV‑cache memory fragmentation.

Flash Attention – Shows how blockwise computation and an online softmax reduce attention memory consumption from O(N²) to O(N).

Stage 6 – Frontier Architectures – Introduces Mixture of Experts (MoE) and Harness Engineering as advanced topics that underpin contemporary large‑model designs.

Each stage is delivered as an independent blog post linked from the repository’s README. The material includes detailed numeric derivations, Python code snippets, and visual explanations, ensuring that no analytical steps are omitted.

Repository

GitHub: https://github.com/amitshekhariitbhu/llm-internals