Beginner’s Guide to Visual Language Models – Day 3: Contrastive Learning Loss Functions

In contrastive learning, loss functions define the objective that shapes the embedding space and ultimately determines how well a model captures meaningful relationships in data.

Contrastive Loss

Originally proposed by Chopra et al. (2005) in Learning a Similarity Metric Discriminatively, with Application to Face Verification , the goal is to pull similar sample pairs together while pushing dissimilar pairs apart by at least a margin m. For a pair (x_i, x_j) with binary label y\in{0,1}:

If the pair is similar ( y=0), minimize the distance D = \|f(x_i)-f(x_j)\|.

If the pair is dissimilar ( y=1), maximize the distance up to the margin m.

Core advantages: efficient discrimination for face verification, image retrieval, sentence embedding, and multimodal alignment.

Triplet Loss

Introduced by Schroff et al. (2015) in FaceNet: A Unified Embedding for Face Recognition and Clustering , the loss processes an anchor a, a positive p (same class), and a negative n (different class). The objective enforces:

\|f(a)-f(p)\|_2^2 + \alpha \le \|f(a)-f(n)\|_2^2

where \alpha is a margin hyper‑parameter. It excels in fine‑grained tasks such as face verification, person re‑identification, and product image search. The main challenge is selecting informative (semi‑hard) negatives; overly easy negatives provide no learning signal, while overly hard negatives can destabilize training.

N‑pair Loss

Proposed by Sohn (2016) in Improved Deep Metric Learning with Multi‑class N‑pair Loss Objective , this loss extends triplet loss by using one anchor, one positive, and N‑1 negatives from the same batch. The model maximizes similarity between anchor and positive while minimizing similarity to all negatives via a softmax‑based formulation, yielding richer gradient signals, more efficient batch training, and better discriminative embeddings.

InfoNCE

First described by van den Oord et al. (2018) in Representation Learning with Contrastive Predictive Coding , InfoNCE treats contrastive learning as a classification problem: given an anchor, the model must identify the positive among N candidates. The loss is:

where sim(a,p) is typically cosine similarity, \tau is a temperature scaling factor, and negatives can be drawn from the batch or a memory bank. InfoNCE powers many state‑of‑the‑art self‑supervised models such as SimCLR, MoCo, CLIP, and DINO, offering efficient large‑scale negative sampling, clear probabilistic interpretation, and strong generality across vision, audio, and video domains.

Cross‑Entropy (Logistic) Loss

Although not exclusive to contrastive learning, binary cross‑entropy (logistic loss) models the probability that two inputs are similar. It is simple, interpretable, and works well for Siamese networks, sentence similarity (e.g., Sentence‑BERT), and other pairwise similarity tasks.

Summary

Loss functions are the compass that guides models to learn semantically meaningful representations, whether the data are images, text, or multimodal. Choosing the appropriate loss—Contrastive, Triplet, N‑pair, InfoNCE, or Cross‑Entropy—significantly impacts downstream performance across tasks such as face verification, fine‑grained classification, few‑shot learning, and visual‑language alignment.

Paper: https://cs.nyu.edu/~sumit/research/assets/cvpr05.pdf

Paper: https://arxiv.org/pdf/1503.03832

Paper: https://dl.acm.org/doi/10.5555/3157096.3157304

Paper: https://arxiv.org/abs/1807.03748