Self-Supervised Learning with Gaussian Processes

Self supervised learning (SSL) is a machine learning paradigm where models learn to understand the underlying structure of data without explicit supervision from labeled samples. The acquired representations from SSL have demonstrated useful for many downstream tasks including clustering, and linear classification, etc. To ensure smoothness of the representation space, most SSL methods rely on the ability to generate pairs of observations that are similar to a given instance. However, generating these pairs may be challenging for many types of data. Moreover, these methods lack consideration of uncertainty quantification and can perform poorly in out-of-sample prediction settings. To address these limitations, we propose Gaussian process self supervised learning (GPSSL), a novel approach that utilizes Gaussian processes (GP) models on representation learning. GP priors are imposed on the representations, and we obtain a generalized Bayesian posterior minimizing a loss function that encourages informative representations. The covariance function inherent in GPs naturally pulls representations of similar units together, serving as an alternative to using explicitly defined positive samples. We show that GPSSL is closely related to both kernel PCA and VICReg, a popular neural network-based SSL method, but unlike both allows for posterior uncertainties that can be propagated to downstream tasks. Experiments on various datasets, considering classification and regression tasks, demonstrate that GPSSL outperforms traditional methods in terms of accuracy, uncertainty quantification, and error control.

💡 Research Summary

The paper introduces GPSSL, a self‑supervised learning framework that replaces the conventional reliance on explicitly generated positive (and sometimes negative) sample pairs with a Gaussian Process (GP) prior over the representation function. The authors first motivate the need for such an approach by pointing out that many non‑image domains (time‑series, tabular, text, graphs) lack natural data‑augmentation pipelines, and that existing SSL methods often provide deterministic embeddings without any measure of uncertainty, which hampers out‑of‑distribution reliability.

GPSSL defines a joint mapping f_z : X → ℝ^J and places a zero‑mean GP prior with kernel k(·,·) on this mapping. Because there are no labels, the standard likelihood is replaced by a generalized Bayesian posterior proportional to the prior times exp(−ℓ(Z)), where ℓ(Z) consists only of the variance and covariance regularizers borrowed from VICReg. The invariance term used in VICReg to pull together positive pairs is omitted, as the GP prior itself encourages smoothness and similarity for inputs that are close under the chosen kernel.

To make inference tractable, the authors adopt sparse variational GP techniques. A set of inducing inputs (U_x, U_z) is introduced, and a Gaussian variational distribution q(U_z) is optimized by maximizing an ELBO that mirrors the classic GP ELBO but with the loss ℓ in place of a log‑likelihood. Monte‑Carlo sampling estimates the expected loss, and the KL term regularizes q toward the GP prior.

The paper provides two theoretical connections. First, if the kernel is defined as an indicator of a positive pair (k(x_i, x_j)=1 when x_j is the augmented version of x_i, 0 otherwise), the GP log‑prior reduces to a dot‑product term identical to an invariance loss, showing that the GP prior implicitly supplies the role of positive pairs. Second, when the variance regularizer is replaced by a weaker alternative, the maximizer of the generalized posterior coincides with the first principal component obtained by kernel PCA, establishing GPSSL as a probabilistic, kernel‑based analogue of kernel PCA and linking modern SSL to classic dimensionality‑reduction methods.

Empirical evaluation spans image benchmarks (CIFAR‑10/100), several UCI tabular datasets, and a real‑world time‑series sensor collection. GPSSL consistently outperforms SimCLR, BYOL, and VICReg in classification accuracy, with gains ranging from 1–3 % on images and comparable improvements on non‑image data. Crucially, because the model yields a posterior distribution over embeddings, the authors can compute predictive uncertainties. Calibration experiments demonstrate that GPSSL’s confidence estimates are far better aligned with true error rates than those of deterministic SSL baselines, reducing over‑confidence in out‑of‑distribution scenarios. The choice of kernel influences performance; the squared‑exponential (RBF) kernel works well across tasks, while data‑driven kernels (e.g., nearest‑neighbor graphs) can capture domain‑specific similarity structures.

Limitations are acknowledged. The variational approximation scales with the number of inducing points, making very large datasets computationally demanding. Hyper‑parameter tuning for the kernel and the variance/covariance coefficients remains necessary. The current implementation is single‑GPU focused, and extending to distributed training or to more expressive sparse GP approximations (e.g., stochastic variational inference, random feature expansions) is left for future work.

In conclusion, GPSSL offers a principled way to enforce smooth, informative representations without handcrafted augmentations, while simultaneously providing uncertainty quantification. The paper opens several avenues for further research, including sparse and scalable GP techniques, automatic kernel learning, and application to multimodal or graph‑structured data where traditional SSL struggles.