Key Deep Learning Architectures for Image Captioning: Encoders, Decoders, Attention & Multimodal Models

Image captioning is a deep‑learning application that merges computer‑vision and natural‑language processing to generate concise textual summaries from images.

Core Components

The typical pipeline consists of three parts: an image encoder, a sequence decoder, and a sentence generator, often enhanced with attention or multimodal fusion.

Image Encoder

The encoder treats the input image as a source and produces a feature representation. Most systems adopt a pretrained convolutional neural network (CNN) backbone—such as VGGNet, ResNet or Inception—remove the final classification layer, and use the remaining convolutional layers to extract a feature map that captures low‑level shapes up to high‑level objects.

Sequence Decoder

The decoder converts the image vector into a token sequence describing the picture. It is usually built from an embedding layer followed by stacked LSTM layers. The image encoding initializes the LSTM’s hidden state, and a start‑token begins the generation loop. At each step the decoder predicts the next token, feeds it back as input for the following step, and terminates when an end‑token is emitted.

Sentence Generator

The generator maps the decoder’s output at each position to a probability distribution over the target vocabulary using a linear layer and a softmax. Greedy search selects the highest‑probability word at each step to form the final caption.

Architectural Variants

Almost all captioning systems follow the encoder‑decoder pattern, but several extensions exist.

Encoder‑Decoder

The simplest design connects the image encoder directly to the LSTM decoder, followed by the sentence generator.

Multimodal (Merge) Architecture

Instead of feeding the encoder output into the decoder, the CNN and LSTM run in parallel. Their outputs are combined by a multimodal layer (often a linear + softmax) before the sentence generator. This allows transfer learning for both the visual backbone and a pretrained language model. Empirically, additive fusion yields the best results.

Detection‑Backbone Architecture

Rather than using a classification‑trained CNN, a pretrained object‑detection model provides region proposals and spatial relationships, producing richer image encodings that capture multiple objects and their positions.

Attention‑Based Encoder‑Decoder

Attention modules compute a weighted sum of image features conditioned on the currently generated word, guiding the LSTM to focus on the most relevant visual region. For example, when generating the word “dog” the model attends to the dog, and when producing “curtain” it shifts focus to the curtain.

Transformer Encoder‑Decoder

Transformer models replace the LSTM with self‑attention layers, preserving the encoder‑decoder structure while enabling richer modeling of spatial relationships among objects. Variants encode not only individual targets but also their relative positions (e.g., “under”, “behind”, “next to”).

Dense Captioning

Building on detection, dense captioning generates multiple captions for different image regions, capturing detailed information across the whole scene.

Generation Strategies

Beyond greedy search, beam search keeps several candidate sequences at each step, selecting the combination with the highest overall probability, which often yields higher‑quality captions.

Evaluation Metric

BLEU measures n‑gram overlap between the generated caption and ground‑truth references. For example, a generated caption “a dog on green grass” versus a ground‑truth “dog on grass” yields a 1‑gram BLEU of 3/6 = 0.5.

Conclusion

With advances in computer vision and NLP, modern image‑captioning models can produce results that closely match human performance.