Understanding Transformers: Self‑Attention, Multi‑Head Mechanisms, and Positional Encoding

Transformer is a classic NLP model introduced by Google in 2017, and models like BERT are built on it. It employs a self‑attention mechanism instead of the sequential RNN structure, enabling parallel training and access to global information.

Traditional sequence processing often uses RNNs, where the input is a vector sequence and the output is another vector sequence. RNNs cannot be parallelized, which limits efficiency.

Although CNNs can also process sequences, they consider only a limited context unless many layers are stacked. Self‑attention replaces RNNs for sequence handling, offering full parallelism.

Self‑attention computes similarity between vectors via dot product, forming Q (query), K (key), and V (value) matrices. The attention scores are obtained by scaling the dot product of Q and K, normalizing them, and weighting V.

Multi‑head attention runs several self‑attention operations in parallel (e.g., eight heads). Each head learns different sub‑spaces while keeping the overall matrix size unchanged, and their outputs are concatenated and projected to produce the final representation.

Since self‑attention lacks positional information, Transformers add positional encoding, defined by sinusoidal functions of token position and dimension. This allows the model to distinguish token order.

The encoding block consists of positional encoding, multi‑head self‑attention, and a feed‑forward network (FFN) with two fully‑connected layers (ReLU followed by a linear layer). Residual connections surround each sub‑layer, and LayerNorm is applied after adding the residual.

In the decoder, an additional encoder‑decoder attention layer attends to encoder outputs, and masked multi‑head attention ensures that each position can only attend to earlier positions. The final decoder output passes through a softmax layer to produce token probabilities, and the model can be trained with losses such as CTC or cross‑entropy.

The article concludes the basic theory of Transformers and notes that the next section will explore their applications in image detection and OCR text recognition.