Demystifying the Core Technologies Behind ChatGPT, GPT‑4, and DeepSeek

01 | Transformer: The Starting Point

Understanding large models begins with the Transformer, introduced in the 2017 Google paper Attention Is All You Need . Its core mechanism, Self‑Attention , lets each token attend to every other token, determining how much focus to give based on relevance. For example, when processing the phrase "DeepSeek released a new model, performance improved dramatically," the token "improved" attends more strongly to "performance" and "dramatically" than to "DeepSeek" or "new". Compared with sequential RNNs, Transformers process all tokens in parallel and establish direct connections between any pair of tokens, yielding faster training and better performance. All major LLM families—GPT, LLaMA, DeepSeek—are built on this architecture.

02 | MoE: Scaling Efficiently

Mixture‑of‑Experts (MoE) addresses the inefficiency of dense models that activate every parameter for each token. In a MoE system the model is split into many "experts" specialized for different tasks, and a router selects only the relevant experts for a given input. This selective activation reduces computation dramatically. DeepSeek V3 exemplifies MoE: although it has 671 billion parameters, inference activates only about 370 billion, making its runtime comparable to a much smaller dense model.

03 | Flash Attention: Handling Long Contexts

Standard attention has O(n²) memory complexity, which quickly exhausts GPU memory for long sequences. Flash Attention (2022) reorganizes the computation to keep intermediate results in fast GPU memory hierarchies and discard them as soon as they are no longer needed. This reduces memory usage by an order of magnitude—for a 10 000‑token sequence, memory drops from several hundred gigabytes to a few tens of gigabytes—while also speeding up execution. Today virtually all leading LLMs, including GPT‑4, LLaMA and DeepSeek, adopt Flash Attention.

04 | KV Cache + MTP: Inference Acceleration

During autoregressive generation each new token requires recomputing attention over all previous tokens. KV Cache stores the previously computed Key and Value matrices so they can be reused, similar to reusing a derived formula in a math problem. Frameworks such as vLLM and TensorRT‑LLM rely on KV Cache to achieve several‑fold speedups.

Multi‑Token Prediction (MTP) pushes the idea further by generating multiple tokens in a single forward pass. Instead of 100 inference steps for 100 tokens, a model might need only 40 steps when generating three tokens at a time. Google’s Gemma 4 already incorporates MTP, and the author predicts it will become a standard feature for future LLM inference.

05 | Quantization: Fitting Models on Smaller Hardware

LLMs store parameters as floating‑point numbers (FP32 or FP16), consuming 2–4 bytes each. A 700‑billion‑parameter model therefore requires 140–280 GB of memory, far beyond consumer GPUs. Quantization converts these floats to lower‑precision integers (e.g., FP32→INT8, FP16→INT4), shrinking model size by roughly fourfold per step. After quantization, a 700‑billion‑parameter model can fit into 35 GB (INT8) or even 17.5 GB (INT4), allowing a RTX 4090 (24 GB) to run a 7‑billion‑parameter INT4 model. The trade‑off is a modest 1‑5 % drop in accuracy, which many consider acceptable for the hardware savings. Projects like llama.cpp and the GGUF format rely on quantization to democratize LLM deployment.

06 | CoT + RAG: Improving Reasoning and Knowledge Freshness

Chain‑of‑Thought (CoT) prompting asks the model to “think step‑by‑step,” which improves correctness on complex tasks because intermediate reasoning steps become part of the context. Models such as OpenAI’s o1 and DeepSeek‑R1 are specifically tuned for CoT and excel at mathematics and code generation.

Retrieval‑Augmented Generation (RAG) tackles the hallucination problem—models confidently generating false statements—by attaching an external knowledge base. When a user asks a question, the system first retrieves relevant documents, then feeds both the retrieved text and the query to the model. This hybrid approach lets enterprises use LLMs for up‑to‑date or private‑domain queries (e.g., legal documents, medical records, customer‑service policies) while mitigating stale knowledge and hallucinations.

Conclusion

The rapid evolution of LLM technology—from the 2017 Transformer breakthrough to today’s MoE, Flash Attention, KV Cache, MTP, quantization, CoT, and RAG—means that each component could fill an entire book. When these components are combined, the remaining work is to apply them to specific tasks.