How Hierarchical Multimodal LSTM Boosts Image Captioning Accuracy

ICCV, one of the top conferences in computer vision, featured a paper by Alibaba iDST and collaborators that aims to teach machines to understand images and generate comprehensive textual descriptions.

Joint Vision‑Language Embedding

Recent advances combine computer vision and natural language processing to create visual‑semantic embeddings, mapping images and sentences into a shared vector space for tasks such as image captioning and cross‑modal retrieval.

Limitations of Prior Methods

Earlier approaches could only embed short sentences, producing coarse descriptions that miss details like objects, attributes, background, and context.

Proposed Framework

The authors propose a two‑step framework: first, detect salient regions in an image and generate descriptive phrases for each region; second, concatenate these phrases into a long, detailed sentence.

This requires a model that can embed not only whole sentences but also individual phrases and image regions.

Hierarchical Multimodal LSTM

A hierarchical LSTM is introduced, where the root node represents the entire image or sentence, leaf nodes correspond to words, and intermediate nodes correspond to phrases or image regions. The model jointly embeds images, sentences, regions, and phrases, learning correspondences between them.

Training includes a loss for each phrase‑region pair to minimize their distance in the embedding space.

Experimental Results

Extensive experiments on public datasets (Flickr30K, MS‑COCO, and a newly annotated MS‑COCO‑region dataset) show that the hierarchical model consistently outperforms state‑of‑the‑art methods in both image annotation (captioning) and image search (retrieval), achieving higher recall@K and median rank scores.

Qualitative visualizations demonstrate that the generated phrases are highly descriptive and that the model can accurately associate image regions with corresponding textual phrases.