Transformer Explained: Self-Attention, Positional Encoding & Encoder-Decoder Code

Introduction: The 2017 Breakthrough

In 2017 Google published the paper Attention Is All You Need [1], introducing the Transformer architecture that rapidly became the foundation of modern NLP and AI. The title's marketing flair aside, the paper's core claim is that attention mechanisms alone can replace recurrent neural networks (RNNs) for sequence modeling.

What Transformers Do Not Need — And What They Still Require

Q: If only attention is needed, what is not needed? A: Recurrent neural networks — no step-by-step sequential processing.

Q: Is attention truly the only component? A: No. The Transformer also relies on six key building blocks:

Feed-Forward Network (FFN)

Residual Connection

Layer Normalization

Embedding Layer

Positional Encoding

Softmax Activation

Items 1, 2, 4, and 6 were already established techniques; the Transformer "stands on the shoulders of giants" by combining them with novel attention and positional encoding designs.

1. Self-Attention Mechanism

Traditional RNN attention computes alignment between decoder hidden states and encoder outputs — like a translator consulting a source document. Transformer's self-attention works differently: each token in a sequence attends to all other tokens simultaneously, akin to team members listening to each other and synthesizing a richer view. This shifts attention from "external query" to "internal introspection," building dynamic, context-aware representations within a single sequence.

How Self-Attention Computes

Each input token embedding passes through three linear projections to produce Query (Q), Key (K), and Value (V) vectors. Attention scores are computed as: Attention(Q, K, V) = softmax(QK⁻ / √d_k) V where d_k is the feature dimension. The scaling factor 1/√d_k controls the softmax input range, preventing gradient vanishing and attention saturation.

Multi-Head Attention

Just as convolutional networks use multiple kernels to capture diverse features, self-attention employs multiple parallel linear projections (heads). Each head learns different relational patterns; their outputs are concatenated and projected again. This multi-head self-attention lets the model jointly attend to information from different representation subspaces.

2. Layer Normalization

Normalization stabilizes deep network training and accelerates convergence. For variable-length sequences with padding tokens, Layer Normalization is preferred over batch normalization. It operates per token across all feature dimensions: first standardizing (subtract mean, divide by standard deviation) to achieve zero mean and unit variance, then applying learnable scale (γ) and shift (β) parameters to restore expressive capacity.

3. Positional Encoding

RNNs inherently capture order by sequential processing. Transformers, processing all tokens in parallel, need explicit position information. The solution is positional encoding : each dimension of the embedding vector receives a sinusoidal wave of a distinct frequency — even dimensions use sine, odd dimensions use cosine. This design ensures that attention scores depend only on relative positions, not absolute indices, which is highly beneficial for natural language where relative order matters more than absolute position.

4. Putting It Together: Encoder-Decoder Architecture

Encoder

The encoder stack consists of 6 identical blocks. Each block contains a multi-head self-attention sublayer and a feed-forward sublayer, each wrapped in a residual connection followed by layer normalization (Add & Norm). The EncoderBlock forward pass:

class EncoderBlock(nn.Module):
    def __init__(self, features: int, self_attention_block: MultiHeadAttentionBlock,
                 feed_forward_block: FeedForwardBlock, dropout: float) -> None:
        super().__init__()
        self.self_attention_block = self_attention_block
        self.feed_forward_block = feed_forward_block
        self.residual_connections = nn.ModuleList([ResidualConnection(features, dropout) for _ in range(2)])

    def forward(self, x, src_mask):
        x = self.residual_connections[0](x, lambda x: self.self_attention_block(x, x, x, src_mask))
        x = self.residual_connections[1](x, self.feed_forward_block)
        return x

class Encoder(nn.Module):
    def __init__(self, features: int, layers: nn.ModuleList) -> None:
        super().__init__()
        self.layers = layers
        self.norm = LayerNormalization(features)

    def forward(self, x, mask):
        for layer in self.layers:
            x = layer(x, mask)
        return self.norm(x)

The Encoder processes the entire sequence at once, enabling full GPU parallelism — a key advantage over sequential RNNs.

Decoder

The decoder also has 6 blocks, but each adds a third sublayer: cross-attention where queries come from the decoder, while keys and values come from the encoder output. This allows the decoder to attend to the encoded source representation. The DecoderBlock forward pass:

class DecoderBlock(nn.Module):
    def __init__(self, features: int, self_attention_block: MultiHeadAttentionBlock,
                 cross_attention_block: MultiHeadAttentionBlock, feed_forward_block: FeedForwardBlock,
                 dropout: float) -> None:
        super().__init__()
        self.self_attention_block = self_attention_block
        self.cross_attention_block = cross_attention_block
        self.feed_forward_block = feed_forward_block
        self.residual_connections = nn.ModuleList([ResidualConnection(features, dropout) for _ in range(3)])

    def forward(self, x, encoder_output, src_mask, tgt_mask):
        x = self.residual_connections[0](x, lambda x: self.self_attention_block(x, x, x, tgt_mask))
        x = self.residual_connections[1](x, lambda x: self.cross_attention_block(x, encoder_output, encoder_output, src_mask))
        x = self.residual_connections[2](x, self.feed_forward_block)
        return x

class Decoder(nn.Module):
    def __init__(self, features: int, layers: nn.ModuleList) -> None:
        super().__init__()
        self.layers = layers
        self.norm = LayerNormalization(features)

    def forward(self, x, encoder_output, src_mask, tgt_mask):
        for layer in self.layers:
            x = layer(x, encoder_output, src_mask, tgt_mask)
        return self.norm(x)

The decoder's second attention module is cross-attention, connecting decoder queries to encoder keys/values. Target masking ( tgt_mask) prevents attending to future positions during autoregressive generation.

Reference

[1] Vaswani et al., "Attention Is All You Need", 2017. arXiv:1706.03762 (https://arxiv.org/abs/1706.03762)