Understanding the Transformer Model: Attention, Self‑Attention, and Multi‑Head Mechanisms

Overview

This article explains the Transformer model, a neural machine translation architecture that relies on attention mechanisms to improve training speed and parallelization.

Encoder and Decoder Structure

The model consists of stacked encoder and decoder layers, each containing a self‑attention sub‑layer and a feed‑forward neural network; the decoder also includes an encoder‑decoder attention sub‑layer.

Self‑Attention

Self‑attention allows each token to attend to all other tokens in the input sequence by projecting the token embeddings into Query, Key, and Value vectors (typically 64‑dimensional) and computing scaled dot‑product attention, followed by a softmax to obtain attention weights.

Multi‑Head Attention

Multi‑head attention runs several parallel self‑attention operations with different learned projection matrices, concatenates their outputs, and projects them back to the model dimension, enabling the model to capture information from multiple representation sub‑spaces.

Positional Encoding

Since the model contains no recurrence, positional encodings are added to token embeddings to inject order information; they are generated using sinusoidal functions as described in the original paper (see get_timing_signal_1d() ).

Residual Connections and Layer Normalization

Each sub‑layer is wrapped with a residual connection and layer‑normalization, facilitating gradient flow and stable training.

Decoder Operation

During decoding, self‑attention is masked to prevent attending to future positions, and the encoder‑decoder attention uses the encoder outputs as Keys and Values.

Final Linear and Softmax Layers

The decoder output is projected through a linear layer to logits over the target vocabulary and a softmax converts these logits into probability distributions for word selection.

Training

Training minimizes a loss such as cross‑entropy between the predicted probability distribution and the ground‑truth tokens, using back‑propagation to update all model parameters.

Inference Strategies

At inference time, greedy decoding or beam search can be employed to generate translations from the probability distributions.