Homomorphic Self-Supervised Learning
AuthorsThomas Anderson Keller, Xavier Suau, Luca Zappella
Homomorphic Self-Supervised Learning
AuthorsThomas Anderson Keller, Xavier Suau, Luca Zappella
This paper was accepted at the workshop “Self-Supervised Learning - Theory and Practice” at NeurIPS 2022.
Many state of the art self-supervised learning approaches fundamentally rely on transformations applied to the input in order to selectively extract task-relevant information. Recently, the field of equivariant deep learning has developed to introduce structure into the feature space of deep neural networks, specifically with respect to such input transformations. In this work, we observe both theoretically and empirically, that through the lens of equivariant representations, many existing self-supervised learning algorithms can be both unified and generalized. Specifically, we introduce a general framework we call Structured Self-Supervised Learning (S-SSL), and theoretically show how it may subsume the concept of input-augmentations provided a sufficiently structured representation. We validate this theory experimentally for simple augmentations, demonstrate how the framework fails when representational structure is removed, and further empirically explore how the parameters of this framework relate to those traditional augmentation-based self-supervised learning. We conclude with a discussion of the potential benefits of this framework as a new perspective on self-supervised learning.
Self-Supervised Learning with Gaussian Processes
January 30, 2026research area Methods and Algorithms
Self supervised learning (SSL) is a machine learning paradigm where models learn to understand the underlying structure of data without explicit supervision from labeled samples. The acquired representations from SSL have demonstrated useful for many downstream tasks including clustering, and linear classification, etc. To ensure smoothness of the representation space, most SSL methods rely on the ability to generate pairs of observations that…
Elastic Weight Consolidation Improves the Robustness of Self-Supervised Learning Methods under Transfer
November 15, 2022research area Computer Vision, research area FairnessWorkshop at NeurIPS
This paper was accepted at the workshop “Self-Supervised Learning - Theory and Practice” at NeurIPS 2022.
Self-supervised representation learning (SSL) methods provide an effective label-free initial condition for fine-tuning downstream tasks. However, in numerous realistic scenarios, the downstream task might be biased with respect to the target label distribution. This in turn moves the learned fine-tuned model posterior away from the initial…