How Attention Mechanisms Revolutionized Computer Vision and Machine Translation

First Application of Attention in Computer Vision

Traditional convolutional models process an entire image at once, which (1) dilutes features of small but important objects and (2) requires high‑resolution computation over the whole frame. In June 2014 DeepMind introduced Recurrent Models of Visual Attention , a model that couples a recurrent neural network (RNN) controller with a hard‑attention sampler. The workflow is:

The RNN receives a low‑resolution global feature map and predicts an initial attention location.

An attention window extracts a high‑resolution patch centered at the predicted location; features from this patch are fed back to the RNN.

The RNN updates its hidden state and predicts the next location.

Steps 2‑3 repeat for a small number of iterations (typically 5–12), allowing the model to converge on the most informative region (e.g., digit contours or object cores).

The sampler implements hard attention : it selects a concrete spatial region rather than assigning soft weights to every pixel. Pixels inside the window are processed at high resolution, while the rest of the image is kept at a coarse resolution, reducing computational cost.

First Application of Attention in Machine Translation

The 2014 paper Neural Machine Translation by Jointly Learning to Align and Translate (Bahdanau, Cho, Bengio) extended the RNN Encoder–Decoder architecture with a soft‑attention alignment module. The encoder compresses a source sentence into a fixed‑size vector C. This vector becomes a bottleneck for long sequences because early‑sentence information is overwritten as the encoder processes more tokens.

Soft attention replaces the static vector C with a dynamic context computed at each decoding step:

For each target token, the decoder computes a similarity score between its current hidden state and each encoder hidden state.

Scores are normalized with a softmax to obtain attention weights.

The weighted sum of encoder states forms a context vector that directly influences the next target token prediction.

This mechanism allows the decoder to focus on the most relevant source words, eliminating the fixed‑size bottleneck and improving translation of long sentences.

Attention in the Transformer Architecture

Before 2017, sequence‑to‑sequence models relied on recurrent networks (LSTM/GRU). Two fundamental limitations motivated a new design:

Sequential computation : each time step depends on the previous hidden state, preventing parallelism and leading to long training times.

Difficulty capturing long‑range dependencies : gradients vanish or explode, and even gated RNNs struggle with sequences longer than a few hundred tokens.

Vaswani et al. removed recurrence entirely in Attention is All You Need . The Transformer introduces:

Positional Encoding : adds a deterministic sinusoidal vector to each token embedding, providing order information without recurrence.

Self‑Attention (scaled dot‑product): for each token, computes attention scores against all other tokens in the same sequence, producing a weighted sum of value vectors. This operation is fully parallelizable.

Cross‑Attention : in the encoder‑decoder configuration, the decoder attends to encoder outputs, replacing the fixed context vector used in earlier RNN models.

Masked Self‑Attention : applied in the decoder; a triangular mask prevents each position from attending to future tokens, preserving causality.

Key empirical results reported in the paper:

English‑German translation BLEU score = 28.4, surpassing the best RNN‑based systems.

Training speed ≈ 4× faster than Google’s GNMT (Google Neural Machine Translation) baseline.

More robust performance on very long sequences, with slower degradation of accuracy.

Subsequent models such as GPT (decoder‑only) and BERT (encoder‑only) adopt the same attention‑only backbone, optionally omitting cross‑attention when the architecture is encoder‑only or decoder‑only. Cross‑attention remains useful in multimodal settings (e.g., vision‑language models) where separate modality encoders must be fused.