Self-Supervised Learning from Structural Invariance

Joint-embedding self-supervised learning (SSL), the key paradigm for unsupervised representation learning from visual data, learns from invariances between semantically-related data pairs. We study the one-to-many mapping problem in SSL, where each datum may be mapped to multiple valid targets. This arises when data pairs come from naturally occurring generative processes, e.g., successive video frames. We show that existing methods struggle to flexibly capture this conditional uncertainty. As a remedy, we introduce a latent variable to account for this uncertainty and derive a variational lower bound on the mutual information between paired embeddings. Our derivation yields a simple regularization term for standard SSL objectives. The resulting method, which we call AdaSSL, applies to both contrastive and distillation-based SSL objectives, and we empirically show its versatility in causal representation learning, fine-grained image understanding, and world modeling on videos.

💡 Research Summary

The paper tackles a fundamental limitation of current self‑supervised learning (SSL) methods: they assume that paired samples (e.g., two augmented views of the same image) share a single, deterministic semantic factor, which leads to a simple, often isotropic noise model for the conditional distribution p(z⁺|z). In real‑world data, especially when pairs are obtained from natural processes such as consecutive video frames, the mapping from a latent state z to its counterpart z⁺ can be highly stochastic, multimodal, and heteroscedastic. Existing contrastive objectives (InfoNCE) and distillation‑based methods (e.g., BYOL) therefore discard information that is not shared between the pair, hurting downstream performance.

To address this, the authors introduce a latent variable r that captures the hidden transformation governing the transition from z to z⁺. By factorising p(z⁺|z) = p(r|z)·p(z⁺|z,r), the conditional distribution conditioned on r becomes much simpler (often unimodal Gaussian), while the complexity of the overall transition is absorbed into the distribution of r. Using the chain rule of mutual information, they rewrite I(f(x);f(x⁺)) as I(f(x), r; f(x⁺)) – I(r; f(x⁺) | f(x)). The first term can be maximised with a standard SSL loss, encouraging the model to use r to reduce uncertainty about x⁺ given x. The second term is penalised via a KL‑divergence regulariser, which forces r to contain information that cannot be inferred from f(x) alone. This yields a variational lower bound on the mutual information that is tractable and can be added as a regularisation term to any existing SSL objective.

Two concrete instantiations are proposed:

1. AdaSSL‑V (Variational) – a continuous latent r is inferred by a variational posterior qϕ(r|x, x⁺). An MLP predictor η takes both f(x) and a sampled r to reconstruct f(x⁺). The KL term between qϕ and a simple prior (e.g., isotropic Gaussian) regularises r.

2. AdaSSL‑S (Sparse) – r is modelled as a sparse, possibly discrete code. L1 sparsity and a modular predictor encourage each dimension of r to correspond to a specific transformation (e.g., camera motion, object acceleration). This version yields interpretable factors and can be combined with contrastive losses.

Both variants can be plugged into contrastive frameworks (InfoNCE) or distillation‑based frameworks (BYOL) with minimal changes, requiring only the additional regularisation term.

The authors validate AdaSSL on three fronts:

• Synthetic numerical data – using mixtures of Gaussians to create multimodal p(z⁺|z). AdaSSL‑V accurately recovers the true modes and outperforms standard contrastive learning on both in‑distribution and out‑of‑distribution tests.

• Fine‑grained image classification – on CUB‑200‑2011 and iNaturalist, AdaSSL‑S improves top‑1 accuracy by 3–5 % over strong baselines, and t‑SNE visualisations show more disentangled class clusters. The method also preserves caption‑level details when trained on image‑caption pairs.

• World modeling in video – on DeepMind Lab and Atari, AdaSSL captures stochastic object accelerations that are invisible to baselines. Future‑frame prediction metrics (PSNR, SSIM) improve by 2–3 dB, and downstream policy learning benefits from richer latent dynamics.

Key contributions are: (i) highlighting the inadequacy of isotropic noise assumptions for naturally paired data, (ii) proposing a latent‑variable‑based variational bound that can be added to any SSL loss, (iii) delivering two practical algorithms (AdaSSL‑V and AdaSSL‑S) that consistently outperform existing methods across causal representation learning, fine‑grained vision, and video world modeling.

Limitations include the need to choose the dimensionality and prior of r, sensitivity to the quality of the variational posterior, and modest computational overhead when scaling to very large datasets. Future work could explore more expressive posterior families (e.g., normalising flows), adaptive r‑dimensionality, and efficient sampling strategies.

Overall, the paper presents a compelling, theoretically grounded, and empirically validated framework that extends self‑supervised learning to handle conditional uncertainty and multimodal transformations inherent in real‑world data.