Demystifying the LLM Tech Stack: Tokens, Embeddings, Attention, and Fine‑Tuning

From Token to Transformer: Foundations of Large Model Architecture

When discussing large language models (LLMs) we often hear terms like Token, Embedding, Transformer, Attention, pre‑training, fine‑tuning, RLHF, RAG, and Agent. This article builds a map that connects these concepts, showing how a sentence travels from raw text to a model’s output.

Large Models Process Tokens, Not Characters

The first technical step is tokenization: converting raw text into the smallest units the model can understand—tokens. For Chinese, the sentence "我喜欢人工智能" may be split into "我 / 喜欢 / 人工 / 智能"; for English, the word "unbelievable" may become "un / believable". Tokens are then mapped to integer IDs, e.g.:

我 -> 1024
喜欢 -> 3812
人工 -> 9045
智能 -> 7721

These IDs are just numbers; they lack semantic meaning until they are passed through an embedding layer.

Embedding: Putting Tokens into a Semantic Space

Each token ID is looked up in a large table to retrieve a high‑dimensional vector, e.g. [0.12, -0.47, 0.83, ...]. During training, vectors for tokens that appear in similar contexts move closer together, so "医生" and "医院" become nearby, as do "函数" and "参数" or "利率" and "央行".

Transformer: The Core Skeleton of Large Models

Tokens (now vectors) plus positional encodings are fed into a Transformer, introduced in the 2017 paper Attention Is All You Need . A Transformer consists of stacked layers, each containing:

Self‑Attention, which lets tokens "see" each other.

Feed‑Forward Network, which further transforms each position’s representation.

Early layers capture shallow relations (part‑of‑speech, local collocations); deeper layers learn syntax, coreference, logic, and task‑specific patterns. After many layers, each token’s vector encodes the whole context.

Attention: Where the Model Focuses

Attention determines which other tokens a given token should attend to. For the sentence "小王把钥匙放进抽屉，因为它很小", the pronoun "它" should attend more to "钥匙" than to "抽屉". The mechanism uses three vectors:

Query – what the current token seeks.

Key – what other tokens can provide.

Value – the actual information carried by other tokens.

Similarity between Query and Key yields attention weights that weight the Values. Multi‑Head Attention runs several attention heads in parallel, each focusing on different aspects such as syntax, coreference, temporal order, or code variable dependencies.

Pre‑training: Source of Model Capability

After the architecture is set, the model undergoes massive pre‑training on large text corpora, learning to predict the next token given previous ones. For example, given the prompt "人工智能正在改变", the model may predict "世界", "行业", "教育", or "医疗" as the next token. The prediction error is measured as loss, and the model parameters are updated via back‑propagation over trillions of such steps.

Through this process the model acquires language structure, common‑sense knowledge, domain expertise, code syntax, reasoning steps, dialogue formats, and stylistic patterns.

Instruction Tuning: From Continuation to Following Commands

Pre‑trained models excel at text continuation but may not obey user instructions. Instruction tuning trains the model on many instruction → response pairs, e.g. translating a sentence, explaining code, or summarizing an article. After this stage the model behaves more like an assistant that understands "please do X".

RLHF: Aligning the Model with Human Preferences

Even after instruction tuning, outputs can be stiff, overly long/short, unsafe, or untruthful. Reinforcement Learning from Human Feedback (RLHF) addresses this by:

Generating multiple answers for the same question.

Having human annotators rank the answers.

Training a reward model to predict human preferences.

Using reinforcement learning to fine‑tune the original model toward higher‑reward answers.

This alignment makes the model more polite, reliable, and aligned with user expectations, though it can also cause over‑conservatism or excessive conformity.

Inference Phase: How the Model Generates Answers

During inference, a user input is tokenized, embedded, passed through the Transformer stack, and the model outputs a probability distribution over the next token. For the prompt "请用一句话解释 Transformer。", candidate next tokens might be:

Transformer: 0.32
它: 0.21
一种: 0.18
简单: 0.05

The model samples a token (guided by parameters such as temperature, top‑k, top‑p, and max tokens), appends it to the context, and repeats until the answer is complete. Because generation is sequential, occasional inconsistencies can arise.

Context Window: The Model’s Short‑Term Memory

The model can only attend to a limited number of tokens—the context window. Larger windows allow processing longer documents or multi‑turn dialogues, but exceeding the limit requires truncation, summarization, or retrieval of older content.

Summarize historical dialogue.

Chunk documents and retrieve relevant pieces.

Store knowledge in a vector database.

Include only task‑relevant content in the prompt.

RAG: Retrieval‑Augmented Generation

RAG (Retrieval‑Augmented Generation) solves the problem of outdated or incomplete model knowledge by retrieving external documents at inference time:

Split documents into chunks.

Embed chunks and store them in a vector database.

Embed the user query.

Retrieve the most relevant chunks.

Pass retrieved chunks together with the query to the model.

The model generates an answer grounded in the retrieved material.

Benefits include up‑to‑date information, private knowledge‑base access, reduced hallucinations, and the ability to cite sources.

Agent: From Answering to Executing Tasks

An Agent extends a model with tool‑calling capabilities. Typical abilities include understanding a goal, decomposing steps, invoking tools, observing results, correcting plans, and iterating. For example, given the instruction "帮我分析这个项目为什么测试失败，并尝试修复。", an agent might:

Read test logs.

Locate failing test cases.

Open relevant source files.

Patch the code.

Rerun tests.

Iterate if failures persist.

Summarize the changes.

This turns the system into a loop of model + tool + environment feedback, raising new concerns about permission control and safety.

Why Large Models Hallucinate

From a technical standpoint, hallucinations arise because the training objective is next‑token prediction, not factual verification. Contributing factors include outdated parameters, incomplete knowledge, insufficient user context, stochastic sampling, and the model’s bias toward producing fluent language even when uncertain.

Engineering mitigations include:

Integrating RAG.

Requiring source citations.

Using external tools for fact‑checking.

Applying rule‑based validation on critical answers.

Human review in high‑risk scenarios.

Reducing sampling randomness (lower temperature).

Testing or executing code to verify results.

Conclusion: Large Models Are a System, Not Just a Model

The end‑to‑end chain is: text → tokens → embeddings → Transformer (self‑attention + feed‑forward) → pre‑training → instruction tuning → RLHF → inference (token‑by‑token generation) → RAG / Agent for external knowledge and tool use. Understanding each link helps build reliable, safe, and useful LLM‑powered applications.