What Powers LLMs? Unpacking Transformers, Architectures, and Context Windows

Transformer Architecture

The Transformer, introduced in the 2017 paper Attention is All You Need , replaced RNN/LSTM models by using a self‑attention mechanism that allows every token to attend to every other token in a single forward pass. This enables direct modeling of long‑range dependencies and fully parallel computation, which dramatically reduces training time and permits scaling to billions of parameters.

Embeddings: Convert input tokens into dense vectors.

Positional Encoding: Inject sequence order information because the attention operation itself is order‑agnostic.

Multi‑Head Self‑Attention: Multiple attention heads learn different relational patterns across the sequence.

Feed‑Forward Network: Applies a non‑linear transformation to each token independently after attention.

Encoder‑Decoder vs. Decoder‑Only Architectures

Both families are built on the Transformer but differ in how the encoder and decoder blocks are used.

Encoder‑Decoder

Typical models: T5, BART, early Google Translate systems.

Operation: The encoder consumes the entire input sequence and produces a contextual representation. The decoder then generates the output token‑by‑token, conditioning on both the encoder representation and previously generated tokens.

Use cases: Machine translation, summarization, extractive/question‑answering tasks that require full input comprehension.

Decoder‑Only

Typical models: GPT‑3, GPT‑4, Llama series, Mistral, PaLM, Gemini (partial).

Operation: A single stack of decoder layers receives a prompt and predicts the next token autoregressively. The predicted token is appended to the prompt and the process repeats until a stop condition is met.

Use cases: Open‑ended text generation, chatbots, code generation, creative writing.

Context Window (Input Length)

The context window defines the maximum number of tokens a model can attend to in a single forward pass. It acts as the model’s short‑term memory and directly limits the amount of text that can be processed without truncation.

Why it matters: Exceeding the window forces the model to discard earlier tokens, which can degrade performance on tasks that require long‑range coherence.

Performance impact: Larger windows improve handling of long documents, extended dialogues, and reasoning over extensive background knowledge.

Typical token limits (as of 2024):

Early GPT models: 2,048 tokens

GPT‑3.5: 4,096 (4K) or 16,384 (16K) tokens

GPT‑4: 8,192 (8K), 32,768 (32K), and experimental 131,072 (128K) tokens

Anthropic Claude: up to 200,000 tokens

Open‑source: Llama 2 base 4K, Mistral 8K–32K, newer models pushing beyond 64K

Research to extend the context window focuses on reducing the quadratic attention cost O(n²). Common techniques include:

Rotary Positional Embedding (RoPE) scaling

Sparse or linear‑complexity attention kernels (e.g., FlashAttention, Longformer, Performer)

Improved positional encodings that remain stable for very long sequences

Challenges remain because longer windows increase memory consumption and can cause the “lost in the middle” phenomenon, where the model’s attention degrades for tokens far from the current position.

Practical Guidance

Select the appropriate architecture: Use encoder‑decoder models when the task requires full input understanding (e.g., translation, summarization). Use decoder‑only models for generation‑centric applications.

Prompt engineering with window limits in mind: Split very long inputs into chunks, summarize earlier sections, or place critical information toward the end of the prompt where the model’s attention is strongest.

Validate claimed window sizes: Benchmark the model on your specific workload because real‑world performance may differ from advertised limits.

Monitor frontier developments: Keep an eye on emerging efficient‑attention algorithms and alternative architectures such as state‑space models (e.g., Mamba) that aim to bypass the quadratic bottleneck.