Why Multimodal AI Matters: Data Modes, Tasks, and Key Models (CLIP & Flamingo)

1 Understanding Multimodal

Why multimodal?

Previously, machine learning handled a single data type—text (translation, language modeling), images (object detection, classification), or audio (speech recognition). True intelligence, however, requires multiple modalities; multimodal data and models enable AI to exhibit human‑like vision and hearing. Multimodal models trained on such data are smarter than pure language models.

What are multimodal data modes?

Data modes include text, images, audio, tabular data, etc. One mode can be approximated by another, for example:

Audio can be represented as a mel‑spectrogram image.

Speech can be transcribed to text, but text loses volume, intonation, pauses.

Images can be encoded as vectors, which can be converted to token sequences.

Video is a sequence of images and audio.

Photographing text yields an image.

Tables can be rendered as charts (images).

Graphs and 3D assets.

Digital formats for smell and touch.

All formats can ultimately be expressed as strings of 0s and 1s for model learning. Text and images are the most common modes and are the focus of this article.

Multimodal tasks

Vision‑language tasks split into generation and vision‑language understanding. The boundary is fuzzy because generation also requires understanding.

Generation tasks: output may be single‑modal (text, image, 3D) or multimodal.

Visual Question Answering: point a camera at anything and ask a question.

Image captioning: automatically generate captions and metadata for images, enabling text‑based image retrieval.

Image generation (txt2img): e.g., DALL‑E, Stable Diffusion, Midjourney.

Text generation.

Vision‑language understanding tasks:

Generate captions or metadata for each image; match query text to the most similar caption.

Joint embedding of images and text; query embedding is compared to image embeddings.

Classification: output a predefined class list (e.g., OCR predicts known characters).

Image‑to‑text retrieval: retrieve the most matching text for an image.

Text‑based image retrieval (TBIR) and image retrieval.

2 Multimodal Training Basics

Many multimodal models exist; this article highlights CLIP (2021) and Flamingo (2022). CLIP was the first model to apply zero‑shot and few‑shot learning to image classification; Flamingo’s impact has been likened to the “GPT‑3 moment” for multimodal AI. Although older, they provide a foundation for newer models.

Common components of multimodal models:

Encoders for each data mode that produce embeddings.

A method to align embeddings from different modes into a shared multimodal embedding space.

(For generative models) a language model that generates text conditioned on both text and visual inputs.

CLIP: Contrastive Language‑Image Pre‑training

CLIP’s key contribution is mapping text and image data into a shared multimodal embedding space, simplifying text‑to‑image and image‑to‑text tasks. The resulting image encoder performs strongly on image classification and can be reused for image generation, visual QA, and text‑based image retrieval. CLIP uses natural‑language supervision and contrastive learning.

Before CLIP, most vision‑language models were trained with classifiers or language‑model objectives. The contrastive objective enables CLIP to generalize across many tasks. An image‑captioning example illustrates why contrastive learning benefits CLIP.

CLIP applications

Image classification: a powerful out‑of‑the‑box baseline, fine‑tunable.

Text‑based image retrieval: useful for searching images with textual queries, though performance lags behind state‑of‑the‑art retrieval systems.

Image generation: DALL‑E generates visual objects from prompts and uses CLIP to rank the outputs.

Text generation, visual QA, captioning: CLIP’s image encoder often serves as the backbone for large multimodal models that generate text.

Flamingo: The Dawn of Large Multimodal Models

Unlike CLIP, Flamingo can generate textual responses; it combines a CLIP‑style image encoder with a language model. Flamingo was trained on four datasets: two image‑text pair sets, one video‑text pair set, and one text‑only set, totaling 2.1 B image‑text pairs (about five times larger than CLIP’s data).

Architecture details:

Vision encoder: uses NFNet‑F6 ResNet; trained on the paired datasets.

Text encoder: BERT (instead of GPT‑2).

Both visual and textual embeddings are mean‑pooled before projection into the joint embedding space.

Language model: Chinchilla.

3 Research Directions for LMMs

Incorporate more data modes into a unified joint embedding space.

Instruction‑following fine‑tuning for multimodal models.

Generate multimodal outputs (e.g., charts, equations, simple animations) to complement multimodal inputs, enabling richer explanations such as visualizing RLHF.