From Next Token to Deployable AI: A Comprehensive Overview of Large Model Technology

Opening a chat window and receiving a response in seconds hides a full technical pipeline that spans data collection, model training, inference optimization, and safety governance.

A large model is not a single algorithm or weight file; it is a production chain where each stage solves the bottleneck of the previous one while exposing new challenges.

Data → Representation → Architecture → Training → Post‑training → Inference Model → Distillation/Evolution → RAG Knowledge System → Multimodal Perception → Agent Action → World Model → Serving → Evaluation → Safety & Trust

Phase 1 – Understanding Model Internals (01‑04)

01 Awakening: From Next‑Token Prediction to Generative Intelligence

Core question: Why does predicting the next token become a universal generative system?

The language‑model probability is defined as the joint probability of a sequence, P(x₁,…,xₙ)=∏P(xₜ|x₁,…,xₜ₋₁). This simple objective forces the model to learn grammar, knowledge, reasoning, and code in order to predict accurately, not to “understand”.

Historical progression: ELIZA (rule‑based, 1966) → N‑gram (frequency) → RNN/LSTM (hidden state) → Transformer (global attention). Each generation solved the previous bottleneck: rules could not generalize, statistics could not handle long‑range dependencies, RNNs could not parallelize, and Transformers removed the parallelism barrier at the cost of O(n²) compute.

Key engineering insight: A chat product consists of seven stacked layers: Base Model → SFT (instruction fine‑tuning) → Preference training (RLHF/DPO) → System Prompt → Chat Template → Safety Layer → Tool Runtime. The same base model can behave very differently under different layer configurations.

Generation parameters (temperature, top‑k, top‑p, repetition penalty) control output diversity, but low temperature does not guarantee factual correctness because model uncertainty is intrinsic.

Conclusion: Autoregressive prediction compresses massive text statistics into a foundation for cross‑task generalisation, yet there is no evidence that the model truly “understands” everything.

02 Representation: Tokenizer and Embedding Mechanics

Core question: How is text split into tokens and turned into vectors the model can process?

Tokenizers dictate discretisation. BPE merges the most frequent adjacent character pairs; Unigram starts from a large vocab and removes low‑likelihood tokens; SentencePiece operates on raw text without pre‑tokenisation; byte‑level BPE eliminates out‑of‑vocabulary tokens.

Token count varies dramatically: Chinese sentences require ~1.8× the tokens of English, and low‑resource languages can need >3×, directly affecting context capacity, training FLOPs, and API cost.

Key engineering insight: The embedding matrix is the first layer with shape V×d (e.g., 128256×4096≈5.25×10⁸ parameters). Token IDs are looked up, summed with RoPE positional encodings, and fed to the model. Extending the vocab requires continued embedding training; otherwise new token vectors are random. Weight tying saves ~25 % of parameters. Retrieval embeddings (e.g., text‑embedding‑3) differ from generation token embeddings.

Conclusion: The tokenizer is a critical component that influences training efficiency, context utilisation, multilingual capability, and inference cost.

03 Architecture: How Transformers Compute Contextual Relations

Core question: What happens during a single forward pass of a Transformer?

Attention uses three matrices: Query (Q), Key (K), Value (V). Each token queries other positions via Q, weights V by the softmax of QKᵀ/√dₖ, achieving content‑addressable global relation modelling. Causal masking ensures a token only sees previous tokens, enabling parallel teacher‑forcing during training but requiring serial generation at inference.

Multi‑head attention allows parallel sub‑space focus. Positional encodings evolved from absolute sinusoidal to RoPE (relative rotation), which rotates Q/K vectors so that dot‑products depend only on relative position, supporting long‑context extrapolation. The feed‑forward network (FFN) holds ~2/3 of parameters and stores knowledge; SwiGLU replaces ReLU, RMSNorm replaces LayerNorm, and Pre‑Norm improves training stability.

Parameter estimate: N≈L·(4d²+2d·d_ff)+V·d. Training FLOPs ≈ 6·N·D (D = token count). Attention cost is O(n²·d), FFN cost O(n·d²); short sequences are FFN‑bound, long sequences become attention‑bound.

Conclusion: Attention resolves information‑access bottlenecks; Transformers boost parallel training but do not eliminate the quadratic cost of long sequences, and context length is distinct from effective context utilisation.

04 Evolution: Dense Transformers, MoE, and SSM

Core question: Why do modern models on the same Transformer skeleton exhibit huge efficiency differences?

GQA (Grouped‑Query Attention) shares KV heads across groups, reducing KV cache size from N to N/k, crucial for long‑context deployment. MQA is an extreme case (all Q share one KV) with noticeable quality loss; GQA balances cost and quality.

FlashAttention is an I/O‑aware exact attention implementation that tiles Q/K/V to minimise data movement between HBM and SRAM, yielding 2‑4× speedups with identical results.

Sparse MoE routes each token to the top‑k experts (e.g., top‑2). DeepSeek‑V3 has 671 B total parameters but activates only 37 B per inference, achieving large capacity with low compute. Challenges include load‑balancing loss, token dropping, shared experts, and all‑to‑all communication.

SSM/Mamba replaces the quadratic attention matrix with linear‑time recurrence, reducing complexity from O(n²) to O(n). Mamba’s selective mechanism decides which information to retain, keeping memory usage constant during inference. However, SSM is weaker at global relation modelling, so hybrid architectures (e.g., Jamba alternating attention and SSM layers) are common.

Key engineering insight: Architecture choice should consider activated parameters, token throughput, task cost, latency, and GPU memory, not just total parameter count. Dense models are simple but expensive; MoE reduces inference cost at the expense of complex training communication; long‑context designs (sliding window + global token) suit low‑latency long sequences; hybrids suit ultra‑large capacity scenarios.

Conclusion: Modern efficiency gains stem from how computation is organised to match hardware characteristics (arithmetic intensity, HBM bandwidth, kernel fusion).

Phase 2 – Understanding How Models Are Manufactured (05‑09)

05 Data: What the Model Actually Eats

Core question: How does training data shape a model’s knowledge, abilities, biases, and limits?

Web data is noisy: spam pages, SEO text, duplicates, and machine‑generated content. A typical pipeline has >10 steps: Crawl → HTML extraction → language ID → exact dedup → MinHash/LSH near‑duplicate detection → quality classifier → toxicity/PII filter → domain mixture weighting → benchmark contamination check → tokenisation.

MinHash creates signatures for documents; LSH buckets similar signatures, retaining only one copy when similarity > 0.8.

Data mixture acts as a “switch” for capabilities: more code data ↑ programming ability, more math data ↑ reasoning, more multilingual data ↑ cross‑language ability. Llama‑3 trained on 15 T tokens (far beyond Chinchilla’s 1.4 T recommendation), illustrating that over‑training can improve inference efficiency.

Key engineering insight: Benchmark contamination occurs via three paths: (1) problem statements scraped into training data, (2) answer text from papers scraped, (3) evaluation code from GitHub scraped. Contamination inflates scores; dynamic evaluation mitigates this. Synthetic data (teacher‑generated → validator‑filtered) can amplify specific abilities, but pure synthetic training leads to model collapse—distribution shrinkage and loss of long‑tail knowledge. Real interaction data fuels the “flywheel”; synthetic data acts as an amplifier. Conclusion: Public internet data is not exhausted; the real scarcity is high‑quality, curated, and verifiable data that introduces new capability gradients.

06 Training: Scaling Laws and Distributed Systems

Core question: How do models reliably train trillions of tokens on thousands of GPUs? Pre‑training bookkeeping: a 7 B model’s weights occupy ~14 GB (BF16). Memory usage per GPU includes weights, gradients, optimizer states, and activations (~56 GB), with the optimizer (AdamW) consuming half. Compute estimate: C≈6·N·D . For a 7 B model trained on 15 T tokens, FLOPs ≈ 6.3×10²³. Kaplan et al. (2020) suggest scaling parameters first; Chinchilla (2022) corrects to a 1:20 parameter‑to‑data ratio (e.g., 70 B parameters → 1.4 T tokens). Modern practice favours over‑training—more tokens for better inference efficiency. Precision roadmap: FP32 → FP16 → BF16 (same dynamic range, half precision, no loss scaling) → FP8 (cutting‑edge, requires E4M3/E5M2 formats). Parallelism combines Data Parallelism, Tensor Parallelism, Pipeline Parallelism, Sequence Parallelism, and Expert Parallelism (for MoE). Communication becomes the primary bottleneck. Key engineering insight: ZeRO‑1/2/3 shards optimizer state, gradients, and parameters across GPUs; PyTorch’s FSDP implements ZeRO‑3. Activation checkpointing trades compute for memory. Model FLOPs Utilisation (MFU) of 45‑55 % is healthy; below 50 % indicates excessive communication. Loss spikes or NaNs require checkpoint rollback. Silent data corruption is a hidden failure mode. Conclusion: Parameter count alone does not dictate training difficulty; context length, activation size, optimizer choice, communication patterns, and MoE routing all significantly affect system complexity.

07 Alignment: CPT, SFT, and Preference Optimisation

Core question: How does post‑training turn a base model into a specialised, controllable assistant? The base model only continues text; given “What is AI?” it may continue “What is artificial grass?” Post‑training reshapes the output distribution without changing underlying knowledge. CPT (Continued Pre‑Training) adjusts knowledge distribution for domain adaptation. SFT (Supervised Fine‑Tuning) teaches instruction‑following behaviour using instruction‑output pairs. Loss masking computes loss only on answer tokens, not on prompts. Packing concatenates multiple short samples into one sequence to improve efficiency. LoRA freezes original weights and learns low‑rank updates ΔW=A·B (rank r ≪ d), reducing parameter count from d² to 2·d·r (> 99 % savings. QLoRA performs LoRA on 4‑bit quantised bases for consumer‑grade GPUs. LoRA changes behaviour but not knowledge capacity. RLHF involves four models: Policy (current), Reference (prevent drift), Reward (scoring), and Value (advantage estimation). KL‑constraint prevents the policy from deviating too far from the reference, mitigating reward hacking. DPO skips the reward and value models, directly optimising on chosen/rejected pairs, increasing good answer probability and decreasing bad answer probability. Simpler and more stable, but unsuitable for tasks requiring exploration. Variants include IPO, KTO, ORPO, SimPO. Key engineering insight: Decision tree for alignment: dynamic knowledge → RAG; behaviour/format → SFT; large domain shift → CPT. Real systems often combine all three. Post‑training must include regression tests to ensure new training does not degrade existing capabilities. Multi‑objective alignment (helpful, honest, harmless, style, domain, cost) creates inherent tension; over‑refusal is a common failure. Conclusion: RLHF is not the sole path to reasoning ability; DPO addresses a different set of engineering constraints.

08 Inference: RLVR, Process Rewards, and Test‑time Scaling

Core question: Does model capability grow from larger weights, reinforcement learning, or more compute at inference? One‑shot generation suffers from local likelihood bias, early error propagation, and hallucination. Chain‑of‑Thought (CoT) makes reasoning steps explicit, but faithfulness is debated—CoT may not reflect true internal reasoning. RLVR (Reinforcement Learning with Verifiable Reward) used in DeepSeek‑R1 automatically grades math problems and runs code, providing binary rewards without human annotation. Rollouts generate multiple paths; a verifier checks correctness, rewarding 1 for correct, 0 otherwise. GRPO (Group Relative Policy Optimisation) uses the group mean as a baseline, rewarding answers above the mean and penalising below‑mean answers, eliminating the need for a separate value model. Reward types: Outcome Reward (final correctness) vs. Process Reward (step‑wise scoring). Process rewards locate error steps but are costlier to label. Reward hacking—models learn to game the verifier rather than solve the problem—is a risk. Test‑time scaling techniques: Best‑of‑N (generate N candidates, pick best), Self‑Consistency (vote among multiple paths), search‑tree + verifier (beam/MCTS). More compute improves accuracy but exhibits diminishing returns (over‑thinking). Key engineering insight: Inference routing: simple tasks → fast model; medium tasks → tool assistance; complex tasks → multi‑path search. Inference latency and cost far exceed those of standard models (e.g., o1/R1 may use 10‑50× more tokens). Verifier design is more important than raw compute. Production systems must not rely solely on visible CoT for audit. Conclusion: Longer thinking does not guarantee correctness; gains depend on model, task, search strategy, and verifier quality.

09 Evolution: Distillation, Synthetic Data, and Self‑Improvement

Core question: Why can models increasingly copy, compress, and expand capabilities? Three growth paths: creation (new data + RL + environment interaction), transfer (distillation from large to small), and release (more inference compute, tools, harnesses without changing the model). Distillation transfers existing ability but cannot create new ability. Distillation types: Response Distillation: teacher generates answer → student learns hard label. Logit Distillation: student learns teacher’s probability distribution (soft target), requiring vocabulary alignment. Reasoning Distillation: teacher provides CoT chain → student learns reasoning steps (used in DeepSeek‑R1). On‑policy Distillation: student generates, teacher critiques, student improves (RL‑like). Synthetic‑data pipeline: Seed → Generate (teacher) → Critique → Verify → Dedup → Difficulty annotation → Curriculum (easy‑to‑hard). Self‑improvement loop: online failure mining → automatic evaluation → data generation → post‑training → regression testing → release. Real‑world failures provide the most valuable training signal. Key engineering insight: Model collapse occurs when pure synthetic data repeatedly trains the model, shrinking distribution and losing long‑tail knowledge. Mitigation: continuously mix real data and environment interactions. Teacher ceiling limits student performance to the teacher’s capacity and verifier quality. Closed‑API models that hide logits restrict response distillation. Conclusion: Distillation mainly moves existing ability; it cannot surpass the combined ceiling of teacher model and verifier.

Phase 3 – Understanding How Models Become Systems (10‑14)

10 Knowledge: RAG, Context Engineering, and Long‑Term Memory

Core question: How to organise knowledge beyond model parameters into high‑quality context? Model parameters have a knowledge cutoff; RAG first retrieves evidence from an external knowledge base, then injects it into the prompt. Simply stuffing documents does not guarantee correct usage. Full RAG pipeline: Ingestion (parse/chunk/metadata) → Index (dense vectors + BM25 sparse + hybrid) → Query transformation (rewrite/HyDE/multi‑query) → Retrieval + reranking (bi‑encoder → cross‑encoder) → Context processing (dedup/compress/sort/cite/conflict handling/budget) → Generate (evidence‑based) → Cite. Challenges: “Lost in the Middle” (middle information ignored), context poisoning (malicious injection), GraphRAG (entity‑graph retrieval for multi‑hop reasoning). Memory hierarchy: Working Memory (current turn) → Episodic (historical events) → Semantic (facts) → User Profile. Write policy decides when to write or forget. Key engineering insight: RAG error taxonomy (parsing, chunking, retrieval, ranking, context conflict, generation). Embedding alone is insufficient; poor retrieval amplifies generation errors. Permission filtering must happen at retrieval. Token budget forces trade‑offs among system instructions, retrieved evidence, tool results, and history summaries. Choose RAG for dynamic knowledge, SFT for behaviour, long‑context for static documents. Conclusion: RAG does not automatically eliminate hallucinations; retrieval errors, evidence conflicts, and unfaithful generation still cause failures.

11 Perception: VLMs Unifying Text, Images, Audio, Video

Core question: How do multimodal models map different signals into a shared reasoning space? From traditional CV (classification, detection, OCR, captioning, VQA) to general Visual‑Language Models (VLM). Vision Transformers split images into patches (e.g., 14×14 pixels) that become visual tokens. Connectors map visual tokens to LLM space via linear projectors, Q‑Former, cross‑attention, or resamplers (token compression). Fusion strategies: Early Fusion (unified token input), Cross‑Attention (modal interaction), Late Fusion (independent encoding then merge), Native Multimodal (train from scratch). Video adds a temporal dimension (patch × frame → spatio‑temporal sequence). Audio tokens discretise audio. Multimodal pre‑training uses interleaved data (image‑text alternation). Grounding aligns text commands to image locations (bounding boxes, pointing). GUI understanding lets models interpret screen interfaces and predict action coordinates, forming the basis of GUI agents. Key engineering insight: High‑resolution images generate many visual tokens, raising cost (a 4K image can produce thousands of tokens). Document parsing must decide between VLM (flexible but costly) and traditional OCR (cheaper). Multimodal hallucinations include object hallucination, OCR errors, spatial errors, counting errors, and temporal inconsistency. Encoding images as tokens does not equate to full 3‑D physical understanding. Conclusion: Multimodal alignment’s goal is a shared space where modalities are comparable and reasonable; Vision‑Language‑Action (VLA) pushes multimodal models toward actionable agents.

12 Action: Tool Use, MCP, and Agent Runtime

Core question: How does a model become an executable system for long‑duration real tasks? Three system forms: Chat (single response), Workflow (fixed DAG), Agent (dynamic policy). Higher autonomy does not always equal higher business value. Function calling lets the model emit structured tool‑call requests (function name + arguments); the system executes and returns results. Constrained decoding guarantees syntactically valid calls. Tool schemas define interfaces. MCP (Model Context Protocol, Anthropic 2024) standardises tool connection: Host/Client/Server architecture where the server offers Tools, Resources, and Prompts. MCP reduces the adaptation effort from M×N to M+N for M models and N tools. Planning approaches: ReAct (reason‑act alternation), Plan‑and‑Execute (plan then act), Hierarchical decomposition, Dynamic replanning after failures. Runtime state machine: Session → Step → Checkpoint → Retry → Complete. Tool calls must be idempotent; compensation handles rollbacks. Hybrid workflow‑agent: embed an agentic node within a deterministic flow to combine reliability and flexibility. Human gate adds manual approval for critical decisions. Multi‑agent setups (supervisor, peer, blackboard, debate) increase communication cost and are not inherently superior. Key engineering insight: Sandbox and permission model must enforce least‑privilege, read/write isolation, network policies, credential brokers, approval gates, transaction limits, and audit logs from the start. Agent evaluation must consider trajectory quality, tool success, recovery, token/cost efficiency, and safety violations. Cost per successful task (not token price) is the meaningful metric; long‑context KV cache can consume tens of GB for 128 K tokens. Conclusion: Agents are more than “large model + loop”; tiny errors compound exponentially in long tasks, and multiple agents do not guarantee superiority.

13 World: World Models, VLA, and Embodied Intelligence

Core question: How do world models support prediction, planning, simulation, and real‑world action? World models differ from high‑fidelity video generation; realistic video does not guarantee physical correctness. World models answer “what would happen if this action is taken” rather than rendering the next frame. Types: Pixel world models (predict raw pixels, costly), Latent world models (predict abstract state, efficient), JEPA (Joint‑Embedding Predictive Architecture) predicts only relevant representations (LeCun’s approach). Model‑based RL uses a dynamics model to imagine future states, then plans on imagined rollouts (Dreamer series). VLA (Vision‑Language‑Action) combines visual input, language goals, and action tokens; RT‑2 discretises actions into tokens. World model handles prediction; policy model decides actions. Sim‑to‑Real: train in simulation → domain randomisation → deploy → calibrate with real feedback. Domain gap remains the main challenge. Key engineering insight: Autonomous driving and robotics benefit from vertical world models constrained to specific physics and sensors. Video quality metrics (FID/IS) cannot replace policy evaluation. Real action data is scarce; video is easy to generate but pairing actions with outcomes is hard. Safety envelopes must combine deterministic constraints with generative predictions. Conclusion: World models aim to provide a distribution of plausible futures for planning, risk assessment, and action selection, not to predict a single exact future.

14 Deployment: Quantisation, KV Cache, and High‑Performance Inference Services

Core question: Why does the same model cost × more on different inference systems? Inference has two stages: Prefill (parallel prompt processing, latency bottleneck) and Decode (token‑by‑token generation, bandwidth bottleneck). TTFT (time to first token) is set by Prefill; TPOT (time per output token) by Decode. KV Cache stores previously computed K/V pairs to avoid recomputation; cache size grows linearly with sequence length, dominating memory for long contexts. Continuous batching dynamically adds/removes requests of varying lengths, avoiding idle GPU time of static batches. PagedAttention treats KV cache like virtual memory pages, reducing fragmentation and enabling prefix sharing. Quantisation pipeline: FP16 → INT8 (≈2× compression) → INT4 (≈4×) → FP8 (latest). GPTQ/AWQ are post‑training quantisation methods. Quantisation reduces memory 2‑4× but does not guarantee end‑to‑end speedup without matching kernels and hardware support. Speculative decoding uses a draft (small) model to generate multiple tokens, while the target (large) model verifies them in parallel, achieving 2‑3× acceleration without loss. Model routing selects small models for simple queries, large models for complex ones. Prefix/semantic cache reuses KV for identical prefixes. Disaggregated Prefill/Decode can run on separate GPUs. Key engineering insight: Service metrics: TTFT, TPOT, throughput, concurrency, goodput (successful tasks per time), GPU utilisation. Throughput and per‑request latency often conflict—high throughput needs large batches, which increase latency. Cost per successful task is the true cost metric, not token price. Long‑context KV can consume tens of GB; FinOps must balance model choice, routing, quantisation, and caching. Conclusion: Quantisation alone does not guarantee speed; inference optimisation is a trade‑off among latency, throughput, memory, and cost.

Phase 4 – Trustworthy Production (15‑16)

15 Evaluation: From Benchmarks to Production‑grade Evals

Core question: How to judge that a model is genuinely stronger, not just higher on leaderboards? Benchmarks provide standardisation and comparability but suffer from saturation, contamination, over‑fitting, and static distributions that miss real‑world variance. Eval hierarchy: Pre‑training Eval (loss/perplexity) → Capability Eval (MMLU, GSM8K, HumanEval) → Alignment Eval (safety/refusal) → System/Agent Eval (task success, trajectory quality, tool success) → Business Eval (KPIs). LLM‑as‑a‑Judge uses a large model to score outputs but introduces biases: position bias, length bias, self‑preference, style bias. Calibration is required; it is not an “automatic truth machine”. Private task sets predict business impact better than public benchmarks; dynamic updates prevent over‑fitting. Blind testing removes brand bias. Contamination isolation (physically separating training and eval data) is essential. Production replay pipelines (shadow traffic → canary → A/B → continuous regression) enforce release gates: a model must pass private regression before deployment. Eval‑driven development cycles: define failure taxonomy → build dataset → define metric → fix → regression test → release, avoiding reliance on a single average score. Conclusion: LLM‑as‑a‑Judge is a useful tool but requires calibration, sampling, and bias control.

16 Trustworthiness: Safety, Explainability, Governance, and Limits

Core question: How to turn a high‑capability model into a controllable, auditable, responsible production system? Risk expands from “saying the wrong thing” to “doing the wrong thing”. Chat safety focuses on harmful outputs; agent safety adds data‑access, tool‑execution, and external side‑effects. Prompt injection is the biggest threat in the agent era: direct injection (malicious prompt) and indirect injection (malicious content hidden in external documents) exploit the blurred boundary between instruction and data. Data‑supply‑chain attacks include poisoning, backdoors, tool poisoning, and memory poisoning. Defense‑in‑depth stack: input filtering → model policy → tool permissions → runtime monitoring → approval → output checking → audit. Model refusal alone is insufficient; safety must be built into system architecture. Agent permission model layers: Identity → Scope → Credential → Approval → Transaction limit → Audit. Mechanistic interpretability (probe → circuit → feature → sparse auto‑encoder → activation steering) reveals what can be explained but cannot replace external controls. Model editing and unlearning aim to modify or erase specific knowledge, yet side‑effects are hard to control, posing challenges for “right‑to‑be‑forgotten” compliance. Governance artefacts: Model Card / System Card, risk tiering, change management, incident response, audit trail, human accountability. Every layer—model, prompt, tool, knowledge base, permission—can shift system risk; alignment is not permanent. Conclusion: Higher capability does not automatically raise reliability or controllability; safety is an ongoing process rather than a static state.

Conclusion: From Acronyms to Judgment

After reading the sixteen parts, you should be able to locate a technical problem within the model, system, or business layer; recognise which new technique resolves an old bottleneck and what new bottleneck it creates; balance capability, cost, latency, throughput, reliability, and controllability; spot common misconceptions such as “bigger = better”, “long context = memory”, “RAG eliminates hallucination”, or “more agents are stronger”; and ask the right questions when communicating with experts.

《大模型简史》的目标是帮助技术用户建立从模型内部、训练系统到生产应用和可信治理的完整坐标系。

References

Vaswani et al., “Attention Is All You Need”, https://arxiv.org/abs/1706.03762

Kaplan et al., “Scaling Laws for Neural Language Models”, https://arxiv.org/abs/2001.08361

Hoffmann et al., “Training Compute‑Optimal Large Language Models (Chinchilla)”, https://arxiv.org/abs/2203.15556

Ouyang et al., “Training language models to follow instructions (InstructGPT)”, https://arxiv.org/abs/2203.02155

Rafailov et al., “Direct Preference Optimization (DPO)”, https://arxiv.org/abs/2305.18290

Lewis et al., “Retrieval‑Augmented Generation (RAG)”, https://arxiv.org/abs/2005.11401

Kwon et al., “PagedAttention/vLLM”, https://arxiv.org/abs/2309.06180

DeepSeek‑AI, “DeepSeek‑R1”, https://arxiv.org/abs/2501.12948

Shumailov et al., “The Curse of Recursion (Model Collapse)”, https://arxiv.org/abs/2305.17493

Hu et al., “LoRA: Low‑Rank Adaptation”, https://arxiv.org/abs/2106.09685

Dao et al., “FlashAttention”, https://arxiv.org/abs/2205.14135

Gu et al., “Mamba: Linear‑Time Sequence Modeling”, https://arxiv.org/abs/2312.00752

Yao et al., “ReAct: Synergizing Reasoning and Acting”, https://arxiv.org/abs/2210.03629

Hinton et al., “Distilling the Knowledge in a Neural Network”, https://arxiv.org/abs/1503.02531