Understanding Seq2Seq: Framework, Advantages, and Decoding Techniques

Scenario Description

As biological organisms we constantly receive sequential visual and auditory signals that the brain interprets, and we also produce sequential outputs when speaking, typing, or driving. In internet services, many data types—text, speech, video, click streams—are sequential, making effective sequence modeling a key research focus.

Problem Description

What is the Seq2Seq framework and what are its advantages?

What common methods are used during Seq2Seq decoding?

Background Assumptions

Basic deep‑learning knowledge is assumed.

The intended audience has some experience with RNNs or work in natural language understanding or sequence modeling.

Answer and Analysis

1. What is the Seq2Seq framework and its advantages?

Before Seq2Seq, deep neural networks performed well on tasks with fixed‑length inputs and outputs, using padding when lengths varied slightly. However, many important problems—machine translation, speech recognition, dialogue generation—produce sequences of unknown length, prompting the development of the Seq2Seq framework around 2013.

The core idea of Seq2Seq is to map an input sequence to an output sequence via two stages: an encoder that reads the input and a decoder that generates the output. In classic implementations both encoder and decoder are recurrent neural networks (RNN, LSTM, or GRU) trained jointly.

In machine translation, the source sentence (e.g., words A B C) is encoded, and the decoder generates the target sentence word by word until an token appears. Similar patterns apply to text summarization (long text → short summary), image captioning (visual features → caption), and speech recognition (audio → transcript).

2. Common decoding methods

The most basic decoding method is greedy search, which selects the highest‑scoring token at each step. It is computationally cheap but only yields a locally optimal solution.

Beam search improves on greedy decoding by keeping the top‑b hypotheses at each step. For example, with beam size b=2, the decoder maintains two partial sequences, expands each with possible next tokens, scores the resulting candidates, and retains the best two for the next step. When b=1, beam search reduces to greedy decoding. Larger beam sizes explore a wider search space and often achieve better translation or summarization quality, at the cost of increased computation (typical values are b≈8–12).

Other decoding enhancements include stacked RNNs, dropout, residual connections between encoder and decoder, attention mechanisms (which let the decoder focus on relevant encoder states at each step), and memory networks that incorporate external knowledge.

References

Auli, Michael, et al. "Joint Language and Translation Modeling with Recurrent Neural Networks." EMNLP, 2013.

Cho, Kyunghyun, et al. "Learning phrase representations using RNN encoder-decoder for statistical machine translation." EMNLP, 2014.

Sutskever, Ilya, Oriol Vinyals, and Quoc V. Le. "Sequence to sequence learning with neural networks." NIPS, 2014.

Bahdanau, Dzmitry, Kyunghyun Cho, and Yoshua Bengio. "Neural machine translation by jointly learning to align and translate." ICLR, 2015.

Sukhbaatar, Sainbayar, Jason Weston, and Rob Fergus. "End-to-end memory networks." NIPS, 2015.

Next Topic Preview

Attention Mechanism