What Is a Context Window? Explaining LLM Memory Capacity

What Is a Context Window?

The context window is the "brain capacity" of a large language model (LLM); the number of tokens it can attend to at once acts as a ruler for its working memory.

Round 2

When you eat and feel sleepy, you forget things because your context window is too small.

Round 3

The context window is the AI’s "brain capacity" and token count is the measuring scale.

Round 4

As content length grows, the number of pairwise token relationships grows quadratically, so processing time slows dramatically. Extending the window without limits would consume massive electricity, cost, and inference latency.

Round 5

KV Cache is the key technology that enables larger context windows. Without it, a modest increase in context would explode GPU memory and compute.

KV Cache stores the key and value vectors from the attention layers so that when generating each new token, the model does not recompute attention over all previous tokens.

Round 6

Ring Attention is an advanced attention mechanism for Transformers that efficiently handles ultra‑long sequences, often used in LLMs processing millions of tokens for tasks like long‑document analysis or video processing.

Round 7

The Needle‑In‑A‑Haystack (NIAH) benchmark evaluates how well LLMs retrieve relevant information from massive, irrelevant context, quantifying the model’s ability to pick out key details.

Round 8

In standard attention (e.g., RoPE positional encoding), as sequence length increases the attention to middle positions dilutes, causing recall rates for middle tokens to drop to 20‑30% while head and tail tokens may still be recalled above 90%.

Round 9

Economic considerations: extending context windows is a competitive race among AI vendors.

Round 10

KV Cache differs from OS caching: it stores intermediate computation results (key/value embeddings) rather than raw data, enabling reuse during autoregressive generation.

Round 11

All AI companies are pushing for longer context windows because greater memory translates to stronger performance.

Next episode preview: what is Retrieval‑Augmented Generation (RAG) and can a character replicate ChatGPT’s code?