How Attention Mechanisms Transform Seq2Seq Models for Better Translation

Scenario Description

As biological organisms, our vision and hearing continuously receive sequential signals that the brain interprets; similarly, we generate sequential signals when speaking, typing, or driving. In internet services, many data types—text, speech, video, click streams—are also sequential, making effective sequence modeling a key research focus.

Problem Description

Why was the attention mechanism introduced into RNN‑based Seq2Seq models, and why are bidirectional RNNs often used in machine‑translation attention models?

Background Assumption

Basic knowledge of deep learning is assumed.

Answer and Analysis

In a typical Seq2Seq architecture, an encoder RNN transforms an input sequence (e.g., a source‑language sentence) into a single vector, which a decoder RNN then expands into an output sequence (e.g., a target‑language sentence). The current hidden state and the previously generated word together determine the next word.

When the input sequence grows, compressing all information into one vector causes severe performance degradation because early words lose influence on later decoding steps. Simple tricks such as reversing the source sentence or duplicating it can provide modest gains, and LSTM cells mitigate but do not fully solve the problem for very long sequences.

Moreover, Seq2Seq decoders often lose or duplicate parts of the input information because contextual and positional cues are discarded during encoding‑decoding.

Introducing attention allows the decoder to focus on relevant encoder hidden states at each generation step, effectively creating a weighted context vector that captures the most pertinent parts of the input.

Attention Mechanism Details

The encoder still produces a series of hidden states h₁, h₂, …, h_T. For each output step i, the decoder computes a context vector c_i as a weighted sum of all encoder states: c_i = Σ_j α_{ij} h_j The attention weights α_{ij} are not fixed; they are produced by a small neural network that takes the previous decoder hidden state s_{i‑1} and each encoder hidden state h_j as inputs, yielding alignment scores e_{ij} which are then normalized (e.g., with softmax) to obtain α_{ij}.

This mechanism lets the model assign larger weights to input words that are most aligned with the current output word, as illustrated by the weight distribution diagrams.

Illustrative Figures

Applications Beyond Seq2Seq

Attention is a versatile concept that can be implemented in many ways and is applied to tasks beyond language translation, such as image captioning, where the model attends to relevant image regions while generating each word.