content type paperpublished May 2024
Poly-View Contrastive Learning
AuthorsAmitis Shidani, Devon Hjelm, Jason Ramapuram, Russ Webb, Eeshan Gunesh Dhekane, Dan Busbridge
Contrastive learning typically matches pairs of related views among a number of unrelated negative views.
Views can be generated (e.g. by augmentations) or be observed. We investigate matching when there are more than two related views which we call poly-view tasks, and derive new representation learning objectives using information maximization and sufficient statistics.
We show that with unlimited computation, one should maximize the number of related views, and with a fixed compute budget, it is beneficial to decrease the number of unique samples whilst increasing the number of views of those samples.
In particular, poly-view contrastive models trained for 128 epochs with batch size 256 outperform SimCLR trained for 1024 epochs at batch size 4096 on ImageNet1k, challenging the belief that contrastive models require large batch sizes and many training epochs.
Related readings and updates.
Contrasting Multiple Representations with the Multi-Marginal Matching Gap
Learning meaningful representations of complex objects that can be seen through multiple (k≥3k\geq 3k≥3) views or modalities is a core task in machine learning. Existing methods use losses originally intended for paired views, and extend them to kkk views, either by instantiating 12k(k−1)\tfrac12k(k-1)21k(k−1) loss-pairs, or by using reduced embeddings, following a one vs. average-of-rest\textit{one vs. average-of-rest}one vs. average-of-rest…
See paper detailsHigh Fidelity 3D Reconstructions with Limited Physical Views
Multi-view triangulation is the gold standard for 3D reconstruction from 2D correspondences, given known calibration and sufficient views. However in practice expensive multi-view setups — involving tens sometimes hundreds of cameras — are required to obtain the high fidelity 3D reconstructions necessary for modern applications. In this work we present a novel approach that leverages recent advances in 2D-3D lifting using neural shape priors…
See paper details
Discover opportunities in Machine Learning.
Our research in machine learning breaks new ground every day.