Search Results for author:
Found 8 papers, 2 papers with code
LoRA+: Efficient Low Rank Adaptation of Large Models
In this paper, we show that Low Rank Adaptation (LoRA) as originally introduced in Hu et al. (2021) leads to suboptimal finetuning of models with large width (embedding dimension).
More is Better in Modern Machine Learning: when Infinite Overparameterization is Optimal and Overfitting is Obligatory
In our era of enormous neural networks, empirical progress has been driven by the philosophy that more is better.
The Effect of SGD Batch Size on Autoencoder Learning: Sparsity, Sharpness, and Feature Learning
On the other hand, for any batch size strictly smaller than the number of samples, SGD finds a global minimum which is sparse and nearly orthogonal to its initialization, showing that the randomness of stochastic gradients induces a qualitatively different type of "feature selection" in this setting.
The Power of External Memory in Increasing Predictive Model Capacity
One way of introducing sparsity into deep networks is by attaching an external table of parameters that is sparsely looked up at different layers of the network.
A Universal Trade-off Between the Model Size, Test Loss, and Training Loss of Linear Predictors
Remarkably, while the Marchenko-Pastur analysis is far more precise near the interpolation peak, where the number of parameters is just enough to fit the training data, it coincides exactly with the distribution independent bound as the level of overparametrization increases.
Deconstructing Distributions: A Pointwise Framework of Learning
In this work, we propose a new approach: we measure the performance of a collection of models when evaluated on a $\textit{single input point}$.
The Three Stages of Learning Dynamics in High-Dimensional Kernel Methods
To understand how deep learning works, it is crucial to understand the training dynamics of neural networks.
Landmark Ordinal Embedding
In this paper, we aim to learn a low-dimensional Euclidean representation from a set of constraints of the form "item j is closer to item i than item k".
