Search Results for author:
Found 28 papers, 10 papers with code
On the low-shot transferability of [V]-Mamba
The strength of modern large-scale neural networks lies in their ability to efficiently adapt to new tasks with few examples.
Theoretical Foundations of Deep Selective State-Space Models
Structured state-space models (SSMs) such as S4, stemming from the seminal work of Gu et al., are gaining popularity as effective approaches for modeling sequential data.
Why do Learning Rates Transfer? Reconciling Optimization and Scaling Limits for Deep Learning
In this work, we find empirical evidence that learning rate transfer can be attributed to the fact that under $\mu$P and its depth extension, the largest eigenvalue of the training loss Hessian (i. e. the sharpness) is largely independent of the width and depth of the network for a sustained period of training time.
SDEs for Minimax Optimization
Minimax optimization problems have attracted a lot of attention over the past few years, with applications ranging from economics to machine learning.
Recurrent Distance Filtering for Graph Representation Learning
Graph neural networks based on iterative one-hop message passing have been shown to struggle in harnessing the information from distant nodes effectively.
Universality of Linear Recurrences Followed by Non-linear Projections: Finite-Width Guarantees and Benefits of Complex Eigenvalues
Deep neural networks based on linear complex-valued RNNs interleaved with position-wise MLPs are gaining traction as competitive approaches to sequence modeling.
Achieving a Better Stability-Plasticity Trade-off via Auxiliary Networks in Continual Learning
In contrast to the natural capabilities of humans to learn new tasks in a sequential fashion, neural networks are known to suffer from catastrophic forgetting, where the model's performances on old tasks drop dramatically after being optimized for a new task.
Resurrecting Recurrent Neural Networks for Long Sequences
Recurrent Neural Networks (RNNs) offer fast inference on long sequences but are hard to optimize and slow to train.
An SDE for Modeling SAM: Theory and Insights
We study the SAM (Sharpness-Aware Minimization) optimizer which has recently attracted a lot of interest due to its increased performance over more classical variants of stochastic gradient descent.
On the Theoretical Properties of Noise Correlation in Stochastic Optimization
This generalizes processes based on Brownian motion, such as the Ornstein-Uhlenbeck process.
Explicit Regularization in Overparametrized Models via Noise Injection
Injecting noise within gradient descent has several desirable features, such as smoothing and regularizing properties.
Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapse
First, we show that rank collapse of the tokens' representations hinders training by causing the gradients of the queries and keys to vanish at initialization.
Anticorrelated Noise Injection for Improved Generalization
Injecting artificial noise into gradient descent (GD) is commonly employed to improve the performance of machine learning models.
On the effectiveness of Randomized Signatures as Reservoir for Learning Rough Dynamics
Many finance, physics, and engineering phenomena are modeled by continuous-time dynamical systems driven by highly irregular (stochastic) inputs.
Faster Single-loop Algorithms for Minimax Optimization without Strong Concavity
Gradient descent ascent (GDA), the simplest single-loop algorithm for nonconvex minimax optimization, is widely used in practical applications such as generative adversarial networks (GANs) and adversarial training.
Rethinking the Variational Interpretation of Accelerated Optimization Methods
The continuous-time model of Nesterov's momentum provides a thought-provoking perspective for understanding the nature of the acceleration phenomenon in convex optimization.
On the Second-order Convergence Properties of Random Search Methods
We prove that this approach converges to a second-order stationary point at a much faster rate than vanilla methods: namely, the complexity in terms of the number of function evaluations is only linear in the problem dimension.
Empirics on the expressiveness of Randomized Signature
Time series analysis is a widespread task in Natural Sciences, Social Sciences and Engineering.
Vanishing Curvature and the Power of Adaptive Methods in Randomly Initialized Deep Networks
This paper revisits the so-called vanishing gradient phenomenon, which commonly occurs in deep randomly initialized neural networks.
Revisiting the Role of Euler Numerical Integration on Acceleration and Stability in Convex Optimization
Viewing optimization methods as numerical integrators for ordinary differential equations (ODEs) provides a thought-provoking modern framework for studying accelerated first-order optimizers.
Two-Level K-FAC Preconditioning for Deep Learning
In the context of deep learning, many optimization methods use gradient covariance information in order to accelerate the convergence of Stochastic Gradient Descent.
Learning explanations that are hard to vary
In this paper, we investigate the principle that `good explanations are hard to vary' in the context of deep learning.
An Accelerated DFO Algorithm for Finite-sum Convex Functions
Derivative-free optimization (DFO) has recently gained a lot of momentum in machine learning, spawning interest in the community to design faster methods for problems where gradients are not accessible.
Practical Accelerated Optimization on Riemannian Manifolds
We develop a new Riemannian descent algorithm with an accelerated rate of convergence.
Optimization and Control
Shadowing Properties of Optimization Algorithms
Ordinary differential equation (ODE) models of gradient-based optimization methods can provide insights into the dynamics of learning and inspire the design of new algorithms.
A Continuous-time Perspective for Modeling Acceleration in Riemannian Optimization
We propose a novel second-order ODE as the continuous-time limit of a Riemannian accelerated gradient-based method on a manifold with curvature bounded from below.
Optimization and Control
The Role of Memory in Stochastic Optimization
We first derive a general continuous-time model that can incorporate arbitrary types of memory, for both deterministic and stochastic settings.
Continuous-time Models for Stochastic Optimization Algorithms
We propose new continuous-time formulations for first-order stochastic optimization algorithms such as mini-batch gradient descent and variance-reduced methods.
