Beginner’s Guide to Visual Language Models – Day 2: Understanding Contrastive Learning

When building visual language models (VLMs), contrastive learning is a key technique that lets the model learn rich representations by pulling together similar samples and pushing apart unrelated ones.

What is contrastive learning?

It treats each image as a point in an embedding space; similar images (e.g., two pictures of a raccoon) are drawn closer, while dissimilar images (e.g., a raccoon and an echidna) are pushed apart. The method does not rely exclusively on labels but exploits intrinsic relationships among samples.

How it works

The process consists of four steps:

Data augmentation – generate different views of the same input through cropping, rotation, color jitter, etc., forming positive pairs.

Encoder processing – feed each view into a CNN or Transformer to obtain feature vectors.

Projection head – map high‑dimensional features to a low‑dimensional space for efficient similarity comparison.

Contrastive loss – minimize distance between positive pairs and maximize distance between negative pairs, usually with cosine similarity.

Selecting positive and negative pairs

Common strategies include Instance Discrimination, where augmented views of the same image are positives and all other images in the batch are negatives, and Image Patching, where patches from the same image form positives while patches from different images are negatives. Typical augmentations are color jitter, rotation, flipping, and noise addition to teach invariance.

Types of contrastive learning

Two main categories are:

Supervised Contrastive Learning (SCL) : uses label information to pull together samples of the same class and push apart samples of different classes, improving class‑aware representations.

Self‑Supervised Contrastive Learning (SSCL) : relies solely on data augmentations; all other images become negatives, which can sometimes separate semantically similar images.

Why contrastive learning matters for VLMs

It enables zero‑shot recognition of unseen categories, supports cross‑modal retrieval (image ↔ text), and improves robustness by learning invariance to visual variations.

Conclusion

The guide promises future posts on loss functions (InfoNCE, Triplet, NT‑Xent) and frameworks such as SimCLR, MoCo, BYOL, and CLIP, showing how the learned representations can power downstream tasks like text‑to‑image generation.