Why Single-Head Attention Falls Short and Multi-Head Saves the Day

Limitations of Single-Head Attention

In self‑attention, each token computes a weighted sum of all other tokens, but a single attention head can only focus on one type of relationship at a time. Consequently, it may attend to the nearest noun or the most salient semantic cue, ignoring other useful signals.

For example, in the sentence "Apple released a new phone because it is tasty," the pronoun it creates an ambiguity: structural proximity suggests "phone," while the word "tasty" points to the fruit "Apple." A single head tends to pick only one of these clues.

Multi‑Head Attention: Multiple Eyes on the Problem

Multi‑head attention introduces several parallel attention heads, each with its own learned projection matrices W_q, W_k, W_v. All heads receive the same input sequence, but because their parameters differ, they compute attention from different perspectives.

Each head produces its own output vector, and the results are concatenated and optionally passed through a final linear layer W_o to fuse the information.

How It Works Internally

Each head receives the same token embeddings.

Because each head has its own W_q, W_k, W_v, it projects the embeddings into a slightly different sub‑space.

The attention scores are computed independently, yielding diverse focus patterns.

The resulting vectors (e.g., four 64‑dimensional vectors for a 256‑dimensional model) are concatenated back to the original dimension.

This design keeps the total computational cost roughly constant while enriching the model’s representational power.

Benefits of Multiple Heads

Capturing Different Ranges: Some heads attend to nearby tokens (short‑range dependencies), others to distant tokens (long‑range dependencies).

Specialized Skills: Empirical studies show heads can become “syntax experts,” “translation experts,” etc.

Robustness: If one head makes a mistake, other heads can compensate, similar to ensemble voting.

However, adding too many heads can lead to redundancy or wasted capacity; practical models choose the number of heads based on overall model size.

Conclusion

Single‑head attention behaves like a single eye with blind spots, while multi‑head attention provides multiple coordinated eyes that jointly capture grammar, semantics, and long‑short dependencies without a significant increase in parameters. The next article will cover positional embeddings, which address the order‑insensitivity of attention.