RNNs, LSTM, and Attention: The Building Blocks Behind LLMs

1. Recurrent Neural Networks

Convolutional neural networks excel in computer vision, while recurrent neural networks (RNNs) are designed for natural language processing (NLP). NLP presents two core challenges: input sequences have variable length, and the same word can carry vastly different meanings depending on its position and context. To process language, a neural network must handle variable-length inputs and possess a "memory" capability to understand semantics from context.

The core of an RNN is its recurrence — a loop between the hidden layer and the input layer. In a typical three-layer setup, each token's embedding vector is fed in one at a time, allowing the network to process sequences of any length. The hidden layer's output, called the hidden state , carries information from previous tokens forward, giving the model a basic ability to use context.

However, this hidden state only provides short-term memory . Tokens that are far apart struggle to influence each other, creating a fundamental limitation for long sequences.

1.1 LSTM

Long Short-Term Memory (LSTM) networks were invented to patch the short-memory problem of vanilla RNNs. The author describes this patch as neither revolutionary nor easy to grasp.

The memory cell state acts as the backbone of LSTM — a horizontal line running through time that stably carries information over long intervals, allowing early inputs to affect later decisions. Maintaining this long-term memory involves two gates:

Forget gate — decides what information to discard from the cell state.

Input gate — decides what new information to add.

The update rule is: New long-term memory = Old long-term memory × Forget gate decision + Input gate decision . The sigmoid activation (output range 0–1) enables this multiplicative gating: values near 1 are kept, values near 0 are forgotten.

The output gate then produces the next short-term memory (hidden state) based on the updated long-term memory.

Because LSTMs are complex and costly to train, a simplified variant called the Gated Recurrent Unit (GRU) was later introduced. The article notes GRU's performance is inferior to LSTM and does not detail it further.

2. Attention Mechanism

In sequence-to-sequence (Seq2Seq) models, the attention mechanism was added to RNNs to improve translation quality for long sentences. At the time, attention was merely a supporting component; few anticipated it would become the central architecture after 2017 (the Transformer paper, arXiv:1706.03762).

The standard encoder-decoder translation flow using RNNs works, and LSTMs can mitigate long-range issues. However, researchers drew an analogy to human translation: we do not memorize an entire sentence at once (encoder); instead, during decoding we repeatedly focus on relevant parts of the source sentence. This insight led to attention.

Implementation is straightforward: compute a weighted sum of the encoder's hidden states at each step to produce an attention vector . In the decoder, this attention vector replaces the simple hidden state, enabling the model to dynamically link to the relevant source segments.

The article omits a PyTorch code example due to hardware constraints.

References: [1] GRU — https://en.wikipedia.org/wiki/Gated_recurrent_unit [2] Seq2Seq — https://en.wikipedia.org/wiki/Seq2seq [3] Transformer (2017) — https://arxiv.org/abs/1706.03762

Code example

1.