content type paperpublished December 2023
Transformers Learn Through Gradual Rank Increase
In collaboration with EPFL, Massachusetts Institute of Technology
AuthorsEnric Boix-Adsera, Etai Littwin, Emmanuel Abbe, Samy Bengio, Joshua Susskind
We identify incremental learning dynamics in transformers, where the difference between trained and initial weights progressively increases in rank. We rigorously prove this occurs under the simplifying assumptions of diagonal weight matrices and small initialization. Our experiments support the theory and also show that phenomenon can occur in practice without the simplifying assumptions.
Related readings and updates.
When Can Transformers Reason With Abstract Symbols?
We investigate the capabilities of transformer models on relational reasoning tasks. In these tasks, models are trained on a set of strings encoding abstract relations, and are then tested out-of-distribution on data that contains symbols that did not appear in the training dataset. We prove that for any relational reasoning task in a large family of tasks, transformers learn the abstract relations and generalize to the test set when trained by…
See paper detailsImplicit Greedy Rank Learning in Autoencoders via Overparameterized Linear Networks
This paper was accepted at the workshop on Overparameterization: Pitfalls and Opportunities at the ICML 2021 conference.
Deep linear networks trained with gradient descent yield low rank solutions, as is typically studied in matrix factorization. In this paper, we take a step further and analyze implicit rank regularization in autoencoders. We show greedy learning of low-rank latent codes induced by a linear sub-network at the autoencoder…
See paper details
Discover opportunities in Machine Learning.
Our research in machine learning breaks new ground every day.