Search Results for author:
Found 8 papers, 2 papers with code
LiDAR: Sensing Linear Probing Performance in Joint Embedding SSL Architectures
In this paper, we introduce LiDAR (Linear Discriminant Analysis Rank), a metric designed to measure the quality of representations within JE architectures.
Vanishing Gradients in Reinforcement Finetuning of Language Models
Pretrained language models are commonly aligned with human preferences and downstream tasks via reinforcement finetuning (RFT), which refers to maximizing a (possibly learned) reward function using policy gradient algorithms.
What Algorithms can Transformers Learn? A Study in Length Generalization
Large language models exhibit surprising emergent generalization properties, yet also struggle on many simple reasoning tasks such as arithmetic and parity.
When can transformers reason with abstract symbols?
We investigate the capabilities of transformer models on relational reasoning tasks.
Adaptivity and Modularity for Efficient Generalization Over Task Complexity
We investigate how the use of a mechanism for adaptive and modular computation in transformers facilitates the learning of tasks that demand generalization over the number of sequential computation steps (i. e., the depth of the computation graph).
The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenon
While common and easily reproduced in more general settings, the Slingshot Mechanism does not follow from any known optimization theories that we are aware of, and can be easily overlooked without an in depth examination.
Implicit Greedy Rank Learning in Autoencoders via Overparameterized Linear Networks
Deep linear networks trained with gradient descent yield low rank solutions, as is typically studied in matrix factorization.
Implicit Acceleration and Feature Learning in Infinitely Wide Neural Networks with Bottlenecks
We analyze the learning dynamics of infinitely wide neural networks with a finite sized bottle-neck.
