Visual Language Model Beginner’s Guide Day 4: Major Contrastive Learning Frameworks

SimCLR – Simple Contrastive Learning of Visual Representations

Paper: https://arxiv.org/abs/2002.05709

SimCLR learns image representations without labels by maximizing similarity between two augmented views of the same image while minimizing similarity with all other images in the batch.

Data augmentation: each image is transformed twice (e.g., random crop, horizontal flip, color jitter, Gaussian blur) to create a positive pair.

Encoder: a backbone such as ResNet‑50 extracts a high‑dimensional feature vector.

Projection head: a small MLP maps the feature to a latent space where the contrastive loss is applied.

NT‑Xent loss: normalized temperature‑scaled cross‑entropy pulls the positive pair together and pushes all other samples apart; large batch sizes provide many negative examples.

Key insight: negative examples are all other images in the batch, which explains why SimCLR benefits from very large batches.

MoCo – Momentum Contrast

Paper: https://arxiv.org/abs/1911.05722

MoCo replaces the need for large batches with a dynamic memory queue and a momentum‑updated encoder.

Dual‑encoder architecture: a query encoder processes the current mini‑batch; a key encoder (an exponential moving average of the query encoder) provides stable historical representations.

Positive‑negative construction: each image yields a query and a key (two augmentations) as a positive pair; negatives are drawn from the queue that stores keys from previous batches.

Momentum update: key encoder weights are updated as θ_k ← m·θ_k + (1‑m)·θ_q with momentum coefficient m, ensuring smooth evolution.

InfoNCE loss: identical to SimCLR, it pulls the query toward its key and pushes it away from all queued negatives.

Core advantage: the queue supplies a large, consistent set of negatives without requiring large batch sizes, improving training stability.

BYOL – Bootstrap Your Own Latent

Paper: https://arxiv.org/pdf/2006.07733

BYOL eliminates negative samples entirely. It trains an online network to predict the representation produced by a target network, both receiving different augmentations of the same image.

Two networks:

Online network = encoder f_θ + projection head g_θ + predictor q_θ.

Target network = encoder f_θ' + projection head g_θ' (no predictor), updated by EMA of the online network.

Training objective: minimize the distance between the online network’s predicted embedding and the target network’s embedding of the other view.

Key insight: BYOL demonstrates that negative samples are not required for high‑quality representation learning; it achieved state‑of‑the‑art ImageNet performance.

SwAV – Swapping Assignments between Views

Paper: https://arxiv.org/abs/2006.09882

SwAV replaces pairwise similarity with a clustering‑based contrastive objective that enforces consistency of cluster assignments across augmentations.

Data augmentation: multiple views are generated as in SimCLR.

Prototype clustering: a set of trainable prototype vectors is maintained; each view predicts a soft assignment via a softmax.

Assignment swapping: the loss forces the assignment of view 1 to match the prediction of view 2 and vice‑versa.

Online codebook optimization: the Sinkhorn‑Knopp algorithm balances assignments across the batch, ensuring efficient learning.

Technical advantage: SwAV requires large batches for the clustering step but achieves ImageNet SOTA with lower GPU/memory usage than SimCLR.

Barlow Twins

Paper: https://arxiv.org/abs/2103.03230

Barlow Twins removes negative samples by explicitly decorrelating feature dimensions. Two augmented views of the same image are passed through a shared encoder, and their cross‑correlation matrix C is forced to approximate the identity matrix.

Workflow:

Generate two augmentations of the same image.

Encode both with a shared backbone (e.g., ResNet‑50) to obtain embeddings z₁ and z₂.

Compute the cross‑correlation matrix C = (z₁ᵀ·z₂) / N, where N is the batch size.

Minimize ∑_i (C_{ii}‑1)² + λ·∑_{i≠j} C_{ij}², driving diagonal elements to 1 (invariance) and off‑diagonal elements to 0 (redundancy reduction).

Key insight: redundancy reduction alone suffices to learn rich, non‑redundant representations without negatives or momentum encoders.

NNCLR – Nearest‑Neighbor Contrastive Learning of Representations

Paper: https://arxiv.org/abs/2104.14548

NNCLR improves SimCLR by selecting the nearest neighbor in a dynamically updated memory bank as the positive sample instead of the other augmentation of the same image.

Positive‑sample selection: each image yields a single augmented view (anchor); its nearest neighbor in the memory bank (based on cosine similarity) becomes the positive.

Encoder & projection: a backbone encoder (e.g., ResNet) and a projection head produce embeddings.

InfoNCE loss: the retrieved neighbor is treated as the positive, all other batch samples as negatives.

Technical advantages: reduces reliance on strong data augmentations, uses a single view per image, and the memory bank evolves to provide higher‑quality neighbors as training progresses.

These six frameworks illustrate distinct strategies for constructing positive and negative pairs, encoder usage, and scalability. SimCLR emphasizes large‑batch augmentation, MoCo introduces a momentum queue, BYOL shows that negatives are unnecessary, SwAV leverages clustering with assignment swapping, Barlow Twins achieves learning through redundancy reduction, and NNCLR incorporates semantic nearest‑neighbor selection. Together they form the technical foundation for many modern self‑supervised vision and vision‑language models.