Search Results for author:
Found 8 papers, 3 papers with code
How Transformers Learn Causal Structure with Gradient Descent
The key insight of our proof is that the gradient of the attention matrix encodes the mutual information between tokens.
Learning Hierarchical Polynomials with Three-Layer Neural Networks
Our main result shows that for a large subclass of degree $k$ polynomials $p$, a three-layer neural network trained via layerwise gradient descent on the square loss learns the target $h$ up to vanishing test error in $\widetilde{\mathcal{O}}(d^k)$ samples and polynomial time.
Fine-Tuning Language Models with Just Forward Passes
Fine-tuning language models (LMs) has yielded success on diverse downstream tasks, but as LMs grow in size, backpropagation requires a prohibitively large amount of memory.
Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stability
Our analysis provides precise predictions for the loss, sharpness, and deviation from the PGD trajectory throughout training, which we verify both empirically in a number of standard settings and theoretically under mild conditions.
Identifying good directions to escape the NTK regime and efficiently learn low-degree plus sparse polynomials
Next, we show that a wide two-layer neural network can jointly use the NTK and QuadNTK to fit target functions consisting of a dense low-degree term and a sparse high-degree term -- something neither the NTK nor the QuadNTK can do on their own.
Increasing Depth Leads to U-Shaped Test Risk in Over-parameterized Convolutional Networks
We then present a novel linear regression framework for characterizing the impact of depth on test risk, and show that increasing depth leads to a U-shaped test risk for the linear CNTK.
Do Deeper Convolutional Networks Perform Better?
Recent work provided an explanation for this phenomenon by introducing the double descent curve, showing that increasing model capacity past the interpolation threshold leads to a decrease in test error.
On Alignment in Deep Linear Neural Networks
We define alignment for fully connected networks with multidimensional outputs and show that it is a natural extension of alignment in networks with 1-dimensional outputs as defined by Ji and Telgarsky, 2018.
