Master AI Fundamentals: Tokens, Context Windows, Temperature, Hallucinations & RAG

1. Introduction

The piece explains five core concepts that anyone using large language models (LLMs) should understand: tokens, context windows, temperature, hallucinations, and retrieval‑augmented generation (RAG). Grasping these ideas helps you write better prompts, control costs, avoid misleading outputs, and leverage advanced AI features.

2. Tokens

LLMs do not read words or letters; they read tokens , which are variable‑length text fragments such as whole words ("cat"), sub‑words ("un", "tion"), or punctuation. For example, the Chinese sentence "我喜欢披萨" is split into three tokens: "我", "喜欢", "披萨". Tokens are the unit of cost: more tokens in a request or response increase processing expense.

3. Context Window

The context window defines how many tokens a model can keep in memory at once. Early models handled about 4,000 tokens; newer models can manage over one million. When the window fills, the oldest tokens are dropped, similar to a whiteboard that must be erased before new writing. This explains why long conversations may lose earlier information.

4. Temperature

The temperature parameter controls output randomness. Low values (near 0) make the model deterministic and safe, always choosing the highest‑probability token—useful for factual tasks like summarization or code generation. High values (≈1 or above) encourage creativity and unexpected word choices—ideal for brainstorming, storytelling, or marketing copy. The article includes a concrete Spring Boot example that sets temperature via ChatOptions.builder().temperature(temperature).build() and calls the model:

@RestController
@RequestMapping("/chat")
public class ChatController {
  private final ChatClient chatClient;
  public ChatController(ChatClient.Builder builder) {
    this.chatClient = builder.build();
  }
  @GetMapping
  public String chat(Double temperature) {
    String userMessage = "完成这个句子：猫坐在……上";
    ChatOptions chatOptions = ChatOptions.builder()
        .temperature(temperature)
        .build();
    Prompt prompt = Prompt.builder()
        .content(userMessage)
        .chatOptions(chatOptions)
        .build();
    return this.chatClient.prompt(prompt)
        .call()
        .content();
  }
}

Two screenshots illustrate low‑temperature (predictable "垫子") versus high‑temperature (varied completions) outputs.

5. Hallucination

Hallucination occurs when an LLM confidently generates false information, presenting fabricated facts as if they were real. Because the model is a statistical predictor rather than a factual database, it will fill gaps with plausible‑looking text instead of saying "I don’t know". This can be dangerous for tasks requiring accurate data, such as medical advice or legal information. The article advises treating AI‑generated facts as starting points that must be verified.

6. Retrieval‑Augmented Generation (RAG)

RAG solves the knowledge‑cutoff problem by retrieving relevant document fragments from a vector database and feeding them to the LLM as additional context. The workflow is:

Upload a document → split into chunks → store embeddings in a vector store.

When a query arrives, perform a similarity search to fetch the most relevant chunks.

Combine the retrieved chunks with the user query and send them to the LLM.

This technique powers many practical AI products—document‑aware chatbots, legal assistants, research summarizers—by augmenting the model’s limited internal knowledge with up‑to‑date external information.

7. Summary

Understanding tokens, context windows, temperature, hallucinations, and RAG equips you to:

Write more effective prompts.

Control cost and output quality.

Avoid blind trust in AI‑generated facts.

Leverage retrieval‑augmented generation for accurate, document‑aware applications.

These five concepts form a practical foundation for anyone working with or evaluating AI systems.