How to Beat Shortcut Learning for Better OOD Generalization in Vision Models

Background

Visual and vision‑language models achieve high accuracy on IID benchmarks but often suffer large performance drops on out‑of‑distribution (OOD) data. The primary cause is shortcut learning : models exploit spurious correlations or superficial visual cues that are predictive in the training set but do not reflect the underlying causal factors.

Why Shortcut Learning Happens

Shortcut learning arises from two intertwined reasons:

Training datasets frequently contain correlations that do not hold in other domains (e.g., static background textures that co‑occur with a specific action).

Gradient‑based optimization tends to converge to flat minima that capture these easy‑to‑learn, non‑causal patterns, because such minima often provide lower loss early in training.

Consequently, models prioritize non‑causal patterns over invariant, robust features, leading to poor OOD generalization.

Two Complementary Remedies

Mitigation can be approached from (a) the data side and (b) the model side :

Data‑level interventions such as targeted augmentations or synthetic data generation break spurious correlations and highlight invariant features.

Model‑level interventions redesign architectures or training objectives to enlarge the basin of minima associated with causal features while suppressing minima linked to shortcuts.

Data‑Level Intervention: StillMix for Video Action Recognition

In video action recognition, static frames often act as shortcuts because they correlate with the action label without containing motion information. StillMix addresses this by mixing a proportion of static frames into each training video while preserving the original action label.

Procedure:

Sample a video clip V = {f_1, …, f_T} and its label y.

Select a set of static frames S = {s_1, …, s_K} from unrelated videos or from the same clip (e.g., duplicated frames).

Replace K randomly chosen positions in V with frames from S, yielding an augmented clip V'.

Feed V' and y to the backbone network as usual.

This augmentation destroys the false correlation between static visual cues and the action label, forcing the network to rely on motion dynamics and improving OOD robustness.

Model‑Level Intervention: FLASH for Few‑Shot Human Action Generation

Few‑shot action generation suffers from over‑fitting to appearance cues because only a handful of examples are available. FLASH combines data augmentation with architectural changes to mitigate this shortcut.

Key components:

Paired video construction : For each training example, generate a partner video that shares the same underlying action but differs in appearance (e.g., by changing background, clothing, or lighting).

Feature alignment loss : Pass both videos through a shared encoder and enforce similarity between their latent representations using a contrastive or L2 alignment term: L_align = || h(V_i) - h(V_i') ||_2^2 Generation objective : The decoder is trained to synthesize realistic motion conditioned on the aligned latent code, encouraging the model to capture motion semantics rather than appearance.

By aligning features across appearance‑variant pairs, FLASH reduces reliance on superficial cues and yields motion representations that generalize to unseen subjects and environments.

Model‑Level Intervention: SPARCL for Vision‑Language Understanding

Vision‑language models often learn coarse visual‑text alignment (e.g., “dog” ↔ any four‑legged animal) and ignore fine‑grained compositional semantics. SPARCL generates synthetic multimodal data with subtle modality variations to force the model to discriminate fine details.

Generation pipeline:

Start from a base image‑text pair (I, T).

Apply controlled perturbations to the image (e.g., change object color, pose, or background) producing I'.

Modify the caption accordingly (e.g., “red dog” vs. “brown dog”) to obtain T'.

Collect a set of such fine‑grained pairs and augment the training corpus.

Training with the augmented set encourages the model to learn compositional representations that capture subtle visual‑text correspondences, thereby improving OOD compositional generalization.

Conclusion

Across three domains—video action recognition, few‑shot human action generation, and vision‑language understanding—data‑level augmentation (StillMix) and model‑level designs (FLASH, SPARCL) demonstrate that explicitly mitigating shortcut learning is essential for robust OOD performance. By breaking spurious correlations and aligning invariant features, these strategies enable visual and multimodal AI systems to learn more causal, transferable representations.