Engineering‑Focused Guide to Training and Inference of Large Language Models

1. Core Mental Model

LLMs fundamentally predict the next token given previous tokens; everything else is designed to make this prediction more accurate, faster, and useful.

Typical data flow:

text → Token → Embedding → Transformer → probability → Token

2. Tokenization and Embedding

Input text is first split into tokens—integer IDs representing sub‑words or characters. Tokens are then mapped to dense embedding vectors that carry semantic information and serve as the model's true input.

Token count directly impacts cost and latency.

Better tokenization improves performance on code and inference tasks.

3. Positional Encoding (RoPE)

Transformers lack inherent order awareness; RoPE (Rotary Positional Encoding) rotates embedding vectors in the vector space to encode relative positions, allowing the model to capture token distance relationships, generalize to long contexts, and is adopted by modern models such as LLaMA.

Engineering insight: RoPE lets the model understand how far apart tokens are rather than just their absolute positions.

4. Self‑Attention: The Core Mechanism

Each token attends to all other tokens, computing similarity scores to aggregate information.

Query: the question a token asks.

Key: what each token contains.

Value: the actual information to be used.

The model calculates attention weights and combines relevant information accordingly.

5. Causal Attention

During generation, a token must not see future tokens. Causal (or masked) attention enforces a strictly left‑to‑right view, making the model autoregressive (one token at a time). Without the mask the model could cheat by looking ahead.

6. Multi‑Head Attention and Variants

Standard Transformers use Multi‑Head Attention (MHA) where each head learns different relationships (syntactic, semantic, long‑range). Variants trade off memory and speed:

MQA (Multi‑Query Attention): all heads share Keys and Values, reducing memory usage and speeding inference.

GQA (Grouped Query Attention): heads are grouped, each group shares Keys and Values, balancing performance and efficiency.

From an engineering standpoint, MHA is powerful but heavyweight; MQA and GQA are production‑optimized alternatives.

7. Transformer Block: Building Block

A Transformer consists of stacked blocks, each containing:

Attention layer

Feed‑Forward Network (FFN)

Residual connection

Layer Normalization

Data flow inside a block:

Input → Attention → Residual → Norm → FFN → Residual → Norm

Residual connections add the block input to its output, stabilizing training and enabling deeper networks. Layer Normalization normalizes activations to keep training stable.

8. Feed‑Forward Network and SwiGLU

After attention, each token passes through an FFN, which processes tokens independently. Modern models replace ReLU with SwiGLU activation, yielding better gradient flow, higher performance, and richer transformations.

Engineering note: Attention gathers information; FFN processes it.

9. Training: From Data to Intelligence

Training starts with pre‑training: predicting the next token on massive corpora using cross‑entropy loss. The model learns language structure, facts, patterns, and basic reasoning.

Key training challenges include distributed system design, GPU utilization, data quality, and memory constraints. Often, higher‑quality data matters more than larger model size.

10. Fine‑Tuning and Alignment

After pre‑training, models are shaped for downstream use:

Supervised Fine‑Tuning (SFT): train on instruction‑response pairs to teach format, style, and behavior.

Instruction Tuning: expose the model to many tasks to improve generalization.

Alignment methods:

RLHF – reinforcement learning from human feedback.

DPO – direct preference optimization (preferred vs. rejected responses).

GRPO – group‑wise preference learning.

Core view: alignment shapes behavior, not knowledge.

11. Parameter‑Efficient Fine‑Tuning

Full‑parameter fine‑tuning is costly. LoRA adds small trainable matrices while freezing the base model, offering low memory usage and fast training. QLoRA combines LoRA with quantization to train large models on modest hardware.

12. Quantization

Quantization reduces precision (e.g., FP16, INT8, INT4) to save memory and accelerate inference, at the cost of slight accuracy loss. Common methods include GPTQ, AWQ, and QLoRA. Quantization is essential for production deployment.

13. Inference

Inference is where the model runs in production. The generation loop is: Input → Predict token → Append → Repeat Key optimizations:

KV Cache: stores intermediate key/value tensors to avoid recomputation, trading memory for speed.

FlashAttention: reduces memory movement during attention calculation.

PagedAttention: manages KV Cache in fixed‑size memory blocks to prevent fragmentation and improve efficiency.

Continuous Batching: dynamically batches incoming requests to maximize GPU utilization.

Speculative Decoding: uses a smaller model to accelerate token generation.

14. Decoding Strategies

The model outputs probabilities; decoding strategies convert them to tokens. Options include Greedy, Sampling, Top‑k, Top‑p, and Temperature, controlling creativity versus determinism.

15. Advanced Inference Models

Techniques such as Chain‑of‑Thought prompting, Self‑Consistency, and Tool Use improve answer quality but increase cost and latency.

16. Toolchain and Practical Workflow

Typical engineering stack:

Hugging Face: model loading, training pipelines, datasets.

Unsloth: faster LoRA/QLoRA training with lower memory footprint.

vLLM: high‑performance inference with PagedAttention, continuous batching, and GPU optimization.

Typical workflow:

Load base model.

Apply LoRA adapters.

Train with Unsloth.

Evaluate.

Export for inference.

Deploy with vLLM.

17. Key Engineering Trade‑offs

Building LLM systems requires balancing:

Accuracy vs. latency.

Memory usage vs. speed.

Cost vs. quality.

Most real‑world work revolves around finding the right compromise.

18. Final Mental Model

An LLM system consists of multiple layers:

Model layer: Attention, Transformer blocks.

Training layer: pre‑training, fine‑tuning, alignment.

System layer: KV Cache, FlashAttention, PagedAttention, batching.

Optimization layer: LoRA, quantization.

Engineers must understand how attention works, how models are trained, how behavior is aligned, and how the serving stack is optimized.