How Transformers Power Modern NLP: A Deep Dive into Encoder‑Decoder Mechanics

Transformer Architecture Overview

Transformers consist of an encoder stack that converts an input token sequence into a rich contextual representation and a decoder stack that generates an output sequence token‑by‑token while attending to the encoder’s output.

Encoder: From Tokens to Contextual Vectors

Input embeddings map each token to a dense vector (e.g., "The" → [0.2, 0.5, -0.1, …]). These vectors encode semantic similarity, syntactic role, and coarse context.

Self‑attention creates three projections for every token: query (Q), key (K) and value (V). For a token i , attention scores are computed as the dot‑product of Q_i with every K_j, scaled by √d_k, and passed through softmax. The weighted sum of the V vectors yields a new representation that mixes information from all positions. Attention(Q,K,V) = softmax(QK^T / sqrt(d_k)) V Example sentence: "The quick brown fox jumps over the lazy dog." The process repeats for each word, allowing the model to capture relationships such as "fox ↔ jumps" or "quick ↔ fast".

Multi‑Head Attention runs the self‑attention mechanism in parallel across 4–8 heads, each with its own learned linear projections. One head may focus on grammatical dependencies, another on positional patterns, and a third on semantic similarity. The outputs of all heads are concatenated and linearly transformed, producing a composite representation that aggregates multiple relational perspectives.

Positional Encoding injects absolute position information because the attention mechanism itself is permutation‑invariant. For position p and dimension i, the encoding is:

PE(p,2i)   = sin(p / 10000^{2i/d_model})
PE(p,2i+1) = cos(p / 10000^{2i/d_model})

Low‑frequency components capture long‑range dependencies, while high‑frequency components emphasize nearby tokens. Adding these vectors to the token embeddings yields a combined vector that encodes both meaning and order.

Feed‑Forward Network (FFN) follows each attention sub‑layer. It consists of two linear layers with a ReLU (or similar) non‑linearity in between, typically expanding the dimensionality from 512 to 2048 and then projecting back to 512. This allows the model to learn higher‑order interactions that pure attention may miss.

Decoder: Autoregressive Generation

The decoder mirrors the encoder but adds two crucial mechanisms:

Masked self‑attention – each position can attend only to earlier positions, preventing the model from seeing future tokens during training or inference.

Encoder‑decoder attention – queries the final encoder representations, grounding the generated output in the source sequence.

After masked self‑attention and encoder‑decoder attention, the decoder applies the same multi‑head attention and FFN blocks as the encoder. The final layer is a linear projection followed by a softmax that yields a probability distribution over the vocabulary, from which the next token is sampled or selected.

Step‑by‑Step Walk‑through

Tokenize the input sentence and look up each token’s embedding.

Add positional encodings to the embeddings.

Pass the sum through N encoder layers, each consisting of:

Multi‑head self‑attention (with Q, K, V projections, scaled dot‑product attention, and concatenation).

Residual connection + layer‑norm.

Feed‑forward network (linear → ReLU → linear).

Residual connection + layer‑norm.

Feed the encoder’s final representations to the decoder.

Masked multi‑head self‑attention over already generated tokens.

Residual + layer‑norm.

Encoder‑decoder multi‑head attention.

Residual + layer‑norm.

Feed‑forward network.

Residual + layer‑norm.

Linear projection + softmax to produce the next token.

Repeat step 4 until an end‑of‑sequence token is emitted.

This iterative stacking of attention and feed‑forward layers enables the model to capture both local and global dependencies, resulting in high‑quality translation, summarisation, and generative text.

Key references: Vaswani et al., "Attention Is All You Need" (2017) https://arxiv.org/abs/1706.03762; original explanatory article https://nintyzeros.substack.com/p/how-do-transformer-workdesign-a-multi