Search Results for author:
Found 7 papers, 3 papers with code
How do Transformers perform In-Context Autoregressive Learning?
More precisely, focusing on commuting orthogonal matrices $W$, we first show that a trained one-layer linear Transformer implements one step of gradient descent for the minimization of an inner objective function, when considering augmented tokens.
Implicit regularization of deep residual networks towards neural ODEs
Our results are valid for a finite training time, and also as the training time tends to infinity provided that the network satisfies a Polyak-Lojasiewicz condition.
Fast, Differentiable and Sparse Top-k: a Convex Analysis Perspective
In this paper, we propose new differentiable and sparse top-k operators.
Vision Transformers provably learn spatial structure
On the theoretical side, we consider a binary classification task and show that while the learning problem admits multiple solutions that generalize, our model implicitly learns the spatial structure of the dataset while generalizing: we call this phenomenon patch association.
Do Residual Neural Networks discretize Neural Ordinary Differential Equations?
As a byproduct of our analysis, we consider the use of a memory-free discrete adjoint method to train a ResNet by recovering the activations on the fly through a backward pass of the network, and show that this method theoretically succeeds at large depth if the residual functions are Lipschitz with the input.
Sinkformers: Transformers with Doubly Stochastic Attention
We show that the row-wise stochastic attention matrices in classical Transformers get close to doubly stochastic matrices as the number of epochs increases, justifying the use of Sinkhorn normalization as an informative prior.
Momentum Residual Neural Networks
We show on CIFAR and ImageNet that Momentum ResNets have the same accuracy as ResNets, while having a much smaller memory footprint, and show that pre-trained Momentum ResNets are promising for fine-tuning models.
Ranked #127 on
Image Classification
on CIFAR-10
