How Transformers Compute Contextual Relationships

1. Why Attention Replaces Recurrence

RNNs suffer from three structural bottlenecks: sequential computation that prevents GPU parallelism, vanishing/exploding gradients over long sequences, and a fixed‑size hidden vector that compresses all input information, causing an information bottleneck. Self‑Attention removes all three by allowing every token to attend to every other token in parallel.

RNN’s three bottlenecks – sequential compute, long‑range gradient decay, and fixed‑size compression – are solved by Transformer’s attention.

2. Core Idea of Attention

Each token generates a Query (what it is looking for), matches it against all Key vectors (what each position has), and aggregates the corresponding Value vectors weighted by similarity. This "query‑retrieve" process replaces the step‑by‑step propagation of RNNs.

3. Q‑K‑V and Scaled Dot‑Product Attention

Given an input matrix X ∈ ℝ^{n×d}, three projection matrices produce:

Q = X·W_Q   (n×d_k)

K = X·W_K   (n×d_k)

V = X·W_V   (n×d_v)

The attention scores are computed as:

Attention(Q, K, V) = softmax(Q·Kᵀ / √d_k) · V

The formula performs four steps: dot‑product similarity, scaling by √d_k, softmax normalization, and weighted sum with V.

4. Causal Mask: Parallel Training vs. Serial Generation

During training, a causal mask sets the upper‑triangular part of the score matrix to –∞, ensuring each position only sees past tokens while allowing the whole sequence to be processed in parallel. During generation, tokens are produced one‑by‑one because each new token’s query depends on the previously generated output.

Training is parallel; generation is serial because of the causal dependency.

5. Multi‑Head Attention

One attention head learns a single pattern; multiple heads learn diverse patterns (local, global, syntactic, semantic). The input is split into h sub‑spaces, each runs Scaled Dot‑Product Attention, and the results are concatenated and projected.

Multi‑Head Attention = parallel sub‑spaces of attention, not an ensemble of models.

6. Positional Encoding

Self‑Attention is permutation‑invariant, so position information must be injected. Original Transformers used sinusoidal encodings; modern models prefer learned embeddings or Rotary Position Embedding (RoPE). RoPE rotates Q and K vectors based on token positions, making attention scores depend on relative distance.

Alternative ALiBi adds a linear bias to the attention scores, offering better extrapolation to very long contexts.

Figure: RNN vs Transformer comparison.

7. Feed‑Forward Network (FFN) and SwiGLU

Each token passes through an FFN: a two‑layer MLP that expands the hidden dimension (typically 4×) before projecting back. FFN parameters account for roughly two‑thirds of a Transformer block’s parameters.

FFN parameters ≈ 2/3 of total block parameters.

SwiGLU replaces ReLU with Swish and adds a gating mechanism, using three weight matrices while keeping parameter count similar.

FFN_SwiGLU(x) = (Swish(x·W₁) ⊙ x·W₃)·W₂

8. Normalization and Residual Connections

Each sub‑layer is wrapped with a residual connection and normalization. Modern models adopt Pre‑Norm (normalization before the sub‑layer) and RMSNorm (variance‑only scaling) for faster training and comparable stability.

Pre‑Norm + RMSNorm are the de‑facto standards in current large language models.

9. One Complete Forward Pass (Decoder‑only example)

Tokenizer converts text to token IDs [t₁,…,tₙ].

Embedding lookup yields X ∈ ℝ^{n×d}.

Positional information is added (RoPE inside attention).

For each of L layers:

Final RMSNorm.

LM head projects to vocabulary size V → logits.

Softmax → probability distribution.

Sampling (greedy / top‑k / top‑p) yields the next token.

Residual Stream = an n×d tensor that carries information through all layers.

10. Parameter and FLOP Estimation

For a model with L layers, hidden size d, FFN intermediate size d_ff, and vocab size V:

N ≈ L·(4d² + 2d·d_ff) + V·d

Training FLOPs are approximated by C ≈ 6·N·D, where D is the number of training tokens (Kaplan et al., 2020; Hoffmann et al., 2022).

Figure: Parameter and FLOPs distribution.

11. Capabilities and Limits

Transformers enable direct token‑to‑token communication, making long‑range dependencies easy to model and speeding up training. However, the quadratic O(n²) attention cost limits context length, prompting research into FlashAttention, sparse/linear attention, and query‑reduction variants (MQA, GQA).

12. Engineering Pitfalls

Deploying a 7 B model with 4 K context runs acceptably, but extending to 32 K can increase latency by ~10× because attention cost grows quadratically and KV‑Cache memory grows linearly. Solutions include switching to GQA/MQA to reduce KV‑head count or using FlashAttention kernels.

Conclusion

Transformer’s core is a combination of Self‑Attention, FFN, residual connections, normalization, and positional encoding. Understanding each component, their parameter/FLOP trade‑offs, and the training vs. inference parallelism boundary is essential for building and deploying large language models.