Search Results for author:
Found 20 papers, 7 papers with code
An SDE for Modeling SAM: Theory and Insights
We study the SAM (Sharpness-Aware Minimization) optimizer which has recently attracted a lot of interest due to its increased performance over more classical variants of stochastic gradient descent.
On the Theoretical Properties of Noise Correlation in Stochastic Optimization
This generalizes processes based on Brownian motion, such as the Ornstein-Uhlenbeck process.
Explicit Regularization in Overparametrized Models via Noise Injection
Injecting noise within gradient descent has several desirable features, such as smoothing and regularizing properties.
Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapse
First, we show that rank collapse of the tokens' representations hinders training by causing the gradients of the queries and keys to vanish at initialization.
Anticorrelated Noise Injection for Improved Generalization
Injecting artificial noise into gradient descent (GD) is commonly employed to improve the performance of machine learning models.
Randomized Signature Layers for Signal Extraction in Time Series Data
Time series analysis is a widespread task in Natural Sciences, Social Sciences, and Engineering.
Faster Single-loop Algorithms for Minimax Optimization without Strong Concavity
Gradient descent ascent (GDA), the simplest single-loop algorithm for nonconvex minimax optimization, is widely used in practical applications such as generative adversarial networks (GANs) and adversarial training.
Rethinking the Variational Interpretation of Accelerated Optimization Methods
The continuous-time model of Nesterov's momentum provides a thought-provoking perspective for understanding the nature of the acceleration phenomenon in convex optimization.
On the Second-order Convergence Properties of Random Search Methods
We prove that this approach converges to a second-order stationary point at a much faster rate than vanilla methods: namely, the complexity in terms of the number of function evaluations is only linear in the problem dimension.
Empirics on the expressiveness of Randomized Signature
Time series analysis is a widespread task in Natural Sciences, Social Sciences and Engineering.
Vanishing Curvature and the Power of Adaptive Methods in Randomly Initialized Deep Networks
This paper revisits the so-called vanishing gradient phenomenon, which commonly occurs in deep randomly initialized neural networks.
Revisiting the Role of Euler Numerical Integration on Acceleration and Stability in Convex Optimization
Viewing optimization methods as numerical integrators for ordinary differential equations (ODEs) provides a thought-provoking modern framework for studying accelerated first-order optimizers.
Two-Level K-FAC Preconditioning for Deep Learning
In the context of deep learning, many optimization methods use gradient covariance information in order to accelerate the convergence of Stochastic Gradient Descent.
Learning explanations that are hard to vary
In this paper, we investigate the principle that `good explanations are hard to vary' in the context of deep learning.
An Accelerated DFO Algorithm for Finite-sum Convex Functions
Derivative-free optimization (DFO) has recently gained a lot of momentum in machine learning, spawning interest in the community to design faster methods for problems where gradients are not accessible.
Practical Accelerated Optimization on Riemannian Manifolds
We develop a new Riemannian descent algorithm with an accelerated rate of convergence.
Optimization and Control
Shadowing Properties of Optimization Algorithms
Ordinary differential equation (ODE) models of gradient-based optimization methods can provide insights into the dynamics of learning and inspire the design of new algorithms.
A Continuous-time Perspective for Modeling Acceleration in Riemannian Optimization
We propose a novel second-order ODE as the continuous-time limit of a Riemannian accelerated gradient-based method on a manifold with curvature bounded from below.
Optimization and Control
The Role of Memory in Stochastic Optimization
We first derive a general continuous-time model that can incorporate arbitrary types of memory, for both deterministic and stochastic settings.
Continuous-time Models for Stochastic Optimization Algorithms
We propose new continuous-time formulations for first-order stochastic optimization algorithms such as mini-batch gradient descent and variance-reduced methods.
