Survey of Lifelong Visual Representations: Continual Self‑Supervised Learning (CSSL)

Motivation for Continual Self‑Supervised Learning

Real‑world visual systems encounter non‑static, unlabeled data streams (e.g., robots entering new environments, remote‑sensing platforms receiving new regions or seasons, vision‑language models facing evolving image‑text distributions). Retraining from scratch for each change is computationally prohibitive, while naïve fine‑tuning on new data risks overwriting previously learned knowledge. Traditional continual learning focuses on supervised class‑incremental tasks with stable labels, which does not reflect these deployment scenarios. Self‑supervised learning (SSL) generates supervision from the data itself, making it a natural fit for continual adaptation.

Paper Contributions

Catalogues common continual self‑supervised learning (CSSL) evaluation protocols, benchmarks, and metrics, highlighting the lack of a unified assessment.

Explains why SSL objectives tend to be more resistant to forgetting than supervised objectives.

Proposes a taxonomy of six CSSL mitigation strategies: distillation, weight regularization, replay, architectural methods, model merging, and objective‑layer adaptation.

Identifies open problems and calls for a shift from small‑scale benchmarks toward large‑scale continual pre‑training.

Self‑Supervised Objectives

Contrastive methods (e.g., SimCLR, MoCo) pull together augmented views of the same image and push apart different images. Non‑contrastive methods avoid explicit negatives through mechanisms such as self‑distillation, stop‑gradient tricks, redundancy reduction, or clustering (e.g., BYOL, SimSiam, Barlow Twins, VICReg, SwAV). Masked‑image modeling (MAE) adapts the NLP masking paradigm to vision by reconstructing masked patches.

In continual settings, contrastive methods require large batches and negative samples, increasing computational cost; non‑contrastive methods are more stable but can still destabilise during continual fine‑tuning; masked modeling fits well with ViT and large‑scale pre‑training but its behaviour under non‑stationary streams remains under‑explored.

Training Protocols

Offline CSSL assumes data arrive as discrete experiences or tasks, each trained for several epochs with clear task boundaries. Evaluation typically occurs after each experience.

Online (boundary‑free) CSSL treats data as a single mini‑batch stream processed only once, without explicit task identifiers. This setting imposes strict compute budgets measured by cumulative backward passes.

Evaluation Metrics and Benchmarks

Representation quality is measured with linear probing and k‑NN accuracy. Continual‑learning metrics include average accuracy, forgetting, forward transfer, and backward transfer. For vision‑language models, average knowledge accumulation and average zero‑shot retention capture the plasticity‑stability trade‑off. Representation stability is quantified with Centered Kernel Alignment (CKA). Common visual benchmarks: CIFAR‑10/100, ImageNet‑100, TinyImageNet. Multimodal continual pre‑training benchmarks: TiC series and FoMo‑in‑Flux.

Why SSL Mitigates Forgetting

More General Representations

Supervised classification compresses features into class‑centric clusters (neural collapse), discarding intra‑class variation needed for future tasks. SSL objectives avoid this collapse, preserving richer, task‑agnostic structure and thus retaining more information across tasks.

Flatter Loss Landscape

Empirical studies report that SSL objectives converge to flatter minima, which are less sensitive to parameter perturbations and therefore more robust to continual updates.

Remaining Challenges

SSL typically requires larger batches, longer training, and momentum encoders, increasing compute.

The stability‑plasticity tension persists: models must retain old knowledge while quickly adapting to new distributions.

Replay buffers can overfit in SSL because unlabeled samples are reused for many epochs; mitigations include mixup, selective sampling, FIFO buffers, or additional stability constraints.

In multimodal settings, continual updates can cause cross‑modal drift, degrading image‑text alignment and zero‑shot performance.

CSSL Mitigation Strategies

Distillation : Preserve similarity between current and previous models in representation space, similarity matrices, or cross‑modal similarity.

Weight Regularization : Limit changes to important parameters (e.g., EWC, SI, MAS). Effective for preventing forgetting but may hinder acquisition of new knowledge in SSL and multimodal contexts.

Replay : Store a subset of old samples and mix them with new data. Straightforward but prone to overfitting in SSL; mitigated by mixup, sample selection, or enhanced stability constraints.

Architectural Methods : Freeze, branch, or isolate parameters, or use low‑rank adapters (e.g., LoRA) to control plasticity. Scalable but risk insufficient plasticity.

Model Merging : Incrementally fuse checkpoints from different time points (e.g., recursive merging in CLIP continual pre‑training) to balance knowledge accumulation and zero‑shot retention.

Objective‑Layer Adaptation : Embed continual‑learning constraints directly into the SSL loss; identified as a promising yet under‑explored direction.

SSL as an Auxiliary Module in Supervised Continual Learning

Even when tasks provide labels, SSL can serve as an auxiliary loss, a pre‑training stage, or a mixed training objective, helping models learn more robust and transferable features. In class‑incremental scenarios, SSL pre‑training mitigates over‑compression of the feature space caused by supervised loss, benefiting domains such as medical imaging, remote sensing, and SAR.

The survey warns that different SSL families (BYOL, SwAV, MAE, DINO, JEPA, etc.) behave differently under non‑stationary streams; therefore, future work must design methods tailored to each SSL objective rather than directly transplanting supervised continual‑learning tricks.

Discussion

Ununified Evaluation Protocols

Works use varied task splits, training budgets, probing methods, and metrics, making fair comparison difficult, especially in online settings where uncontrolled compute can bias results.

From Small Benchmarks to Foundation Models

Most CSSL research remains on CIFAR, TinyImageNet, and ImageNet‑100. Real lifelong systems will face long horizons, massive data, non‑stationary distributions, and multimodal inputs. The paper advocates moving toward continual pre‑training at scale.

Large‑Model and Multimodal Challenges

Scaling up introduces new issues: full model updates become costly, prompting the need for parameter‑efficient updates, model merging, selective freezing, and budget‑aware training. Vision‑language models must handle drift in image‑text alignment and maintain robustness when one modality is missing. Selective forgetting is also emphasized; models should identify obsolete patterns and discard them rather than preserving all old features indefinitely.

Conclusions

Self‑supervised objectives produce more general and stable representations because they avoid task‑centric feature collapse and converge to flatter loss minima. However, SSL alone does not solve continual adaptation; mechanisms such as distillation, replay, architectural strategies, model merging, and especially objective‑layer adaptation remain necessary. Progress depends on unified evaluation, scalable computation, rapid adaptation, cross‑modal drift mitigation, and long‑term representation evolution.

Code example

来源：专知
本文
约4000字
，建议阅读
5
分钟
这篇综述系统梳理了视觉模型中的持续自监督学习。