How Self-Attention Powers Modern AI: From Theory to Real-World Impact

Self‑Attention Overview

Self‑Attention computes a relevance score between every pair of elements in a sequence, producing dynamic attention weights that capture global dependencies. Each token queries the whole sequence and aggregates information from tokens that receive high weights.

Core Elements

Query (Q)

: the vector representing the current token’s “question”. Key (K): the vector representing each token’s “content descriptor”. Value (V): the vector containing the actual information to be passed forward. Attention Score: similarity (dot‑product) between a Query and a Key. Softmax Normalization: converts raw scores into a probability distribution that sums to 1. Weighted Sum: aggregates the Value vectors using the normalized attention weights.

Mathematical Formulation

Input: X (n×d)
Linear projections:
  Q = XW_Q
  K = XW_K
  V = XW_V
Score matrix: S = Q·Kᵀ
Scaled scores: Ŝ = S / √d_k
Attention matrix: A = softmax(Ŝ)   (n×n)
Output: Z = A·V   (n×d)

The scaling factor √d_k prevents the softmax from saturating when the dot‑product magnitude grows with the dimensionality d_k.

Step‑by‑Step Example

Sentence: The animal didn’t cross the street because it was too tired. Compute attention for the pronoun it:

The : similarity 0.02 → weight 1 % (low)

animal : similarity 0.85 → weight 45 % (high)

didn’t : similarity 0.05 → weight 3 % (low)

cross : similarity 0.08 → weight 4 % (low)

street : similarity 0.12 → weight 6 % (low)

because : similarity 0.15 → weight 8 % (medium)

it : similarity 0.50 → weight 26 % (medium)

was : similarity 0.06 → weight 3 % (low)

too : similarity 0.04 → weight 2 % (low)

tired : similarity 0.13 → weight 7 % (low)

The highest weight (45 %) is assigned to animal , so the model resolves the coreference correctly: it refers to the animal, not the street.

Multi‑Head Attention

In practice, several self‑attention heads run in parallel, each learning a different relational pattern. Example configuration:

Head 1: syntax (subject‑verb‑object)
Head 2: semantic similarity (synonyms)
Head 3: positional relations (adjacent tokens)
Head 4: long‑range dependencies (coreference)
…
Head 8: domain‑specific patterns

The outputs of all heads are concatenated, giving the model a richer, multi‑faceted representation.

Computational Cost

The time complexity of self‑attention is O(n²·d), where n is the sequence length and d the hidden dimension. For a 1 000‑token sentence, the model performs 1 000 000 pairwise dot‑products.

Comparison with RNNs and CNNs

Computation pattern : RNN/LSTM – strictly serial; CNN – locally parallel (limited receptive field); Self‑Attention – fully parallel across all positions.

Long‑range dependencies : RNN/LSTM struggle due to gradient vanishing; CNN requires deep stacking; Self‑Attention captures them in a single layer.

Interpretability : RNN/CNN weights are opaque; Self‑Attention weights can be visualized to see which tokens influence each other.

Training speed : RNN – slow; CNN – fast; Self‑Attention – moderate (parallelism offsets quadratic cost).

Key References

Vaswani et al., “Attention Is All You Need”, arXiv:1706.03762 (2017).

Jay Alammar, “The Illustrated Transformer”, jalammar.github.io/illustrated-transformer/ .

Zhihu article “Attention Mechanism Detailed”, zhuanlan.zhihu.com/p/47063917 .

Jesse Vig, “BERTViz – BERT attention visualization tool”, GitHub github.com/jessevig/bertviz .