Why Transformers Outperform RNNs: A Beginner’s Guide to Attention and Architecture

What is a Transformer?

Transformer is a novel architecture that uses attention mechanisms to dramatically improve the performance of deep‑learning NLP translation models. It first appeared in the paper “Attention is all you need” and quickly became the mainstream for processing textual data.

Since then, many projects such as Google’s BERT and OpenAI’s GPT series are built on this architecture, outperforming previous state‑of‑the‑art techniques.

This article, the first in a series, introduces how to use Transformers, why they are superior to RNNs, and describes their architecture components and behavior during training and inference.

Transformer Architecture

The core of a Transformer consists of multiple encoder and decoder layers. A single layer is called an encoder or decoder, and a stack of such layers forms an encoder group or decoder group.

Each encoder group and decoder group has corresponding embedding layers that process their inputs, and the final output layer produces the result.

Encoder and decoder layers share the same structure: the encoder contains a self‑attention sub‑layer and a feed‑forward sub‑layer, while the decoder adds an encoder‑decoder attention sub‑layer. All layers have their own weight matrices and are wrapped with residual connections and two LayerNorm layers.

Some Transformer variants omit the decoder entirely and rely only on the encoder.

How Attention Works

Attention allows the model to focus on words that are closely related to the current word while processing a sequence. For example, “Ball” is closely related to “blue” and “holding” but not to “boy”. Self‑attention computes a weighted sum of all other words for each word, enabling the model to capture these relationships.

Multi‑head attention provides multiple attention scores for each word, allowing the model to consider different aspects such as the word itself and its surrounding context.

The cat drank the milk because it was hungry.

The cat drank the milk because it was sweet.

In the first sentence “it” refers to “cat”; in the second it refers to “milk”. Self‑attention supplies the model with richer contextual information to resolve such ambiguities.

Training a Transformer

During training the model receives both a source (input) sequence and a target sequence. The goal is to learn to generate the target sequence from the source.

Source sequence (e.g., English “You are welcome”).

Target sequence (e.g., Spanish “De nada”).

The training pipeline:

Convert the input sequence into embeddings with positional encoding and feed it to the encoder.

The encoder stack produces an encoded representation of the input.

Add a start‑of‑sentence token to the target sequence, embed it, and feed it to the decoder.

The decoder processes this together with the encoder’s representation to produce a decoded representation.

The output layer converts the decoded representation into word probabilities and the final output sequence.

The loss function compares the output sequence with the ground‑truth target sequence and back‑propagates the error.

Inference

In inference only the input sequence is available. The model generates the output sequence token by token, feeding the partially generated sequence back into the decoder at each step until an end‑of‑sentence token is produced.

Because the encoder’s representation does not change, the encoder steps are performed once, making inference efficient.

Teacher Forcing

During training the target sequence is fed directly into the decoder (teacher forcing). This provides the correct next token as a clue, preventing error accumulation that would occur if the model had to rely on its own predictions.

Teacher forcing also enables parallel computation of all tokens, speeding up training.

Applications of Transformers

Transformers are widely used in NLP tasks such as machine translation, text summarization, question answering, named‑entity recognition, and speech recognition. Different tasks attach specific “head” modules to the Transformer output, e.g., a classification head for sentiment analysis.

Why Transformers Beat RNNs

RNN‑based seq2seq models process tokens sequentially, which limits their ability to capture long‑range dependencies and slows down training and inference. Transformers process all tokens in parallel and compute dependencies regardless of distance, resulting in faster training and better performance.

Next article will dive deeper into the internal workings of the Transformer.

Original source: https://towardsdatascience.com/transformers-explained-visually-part-1-overview-of-functionality-95a6dd460452