Search Results for author:
Found 13 papers, 0 papers with code
Stochastic Gradient Descent outperforms Gradient Descent in recovering a high-dimensional signal in a glassy energy landscape
Stochastic Gradient Descent (SGD) is an out-of-equilibrium algorithm used extensively to train artificial neural networks.
The effective noise of Stochastic Gradient Descent
In the under-parametrized regime, where the final training error is positive, the SGD dynamics reaches a stationary state and we define an effective temperature from the fluctuation-dissipation theorem, computed from dynamical mean-field theory.
Stochasticity helps to navigate rough landscapes: comparing gradient-descent-based algorithms in the phase retrieval problem
In this paper we investigate how gradient-based algorithms such as gradient descent, (multi-pass) stochastic gradient descent, its persistent variant, and the Langevin algorithm navigate non-convex loss-landscapes and which of them is able to reach the best generalization error at limited sample complexity.
Analytical Study of Momentum-Based Acceleration Methods in Paradigmatic High-Dimensional Non-Convex Problems
The optimization step in many machine learning problems rarely relies on vanilla gradient descent but it is common practice to use momentum-based accelerated methods.
Disordered high-dimensional optimal control
In the noisy case one has a set of controllable stochastic processes and a cost function that is a functional of their trajectories.
Optimization and Control Disordered Systems and Neural Networks
Low-frequency vibrational spectrum of mean-field disordered systems
We study a recently introduced and exactly solvable mean-field model for the density of vibrational states $\mathcal{D}(\omega)$ of a structurally disordered system.
Disordered Systems and Neural Networks Soft Condensed Matter Statistical Mechanics
Surfing on minima of isostatic landscapes: avalanches and unjamming transition
This algorithm allows to "surf" between isostatic marginally stable configurations and to investigate some properties of such landscape.
Disordered Systems and Neural Networks Statistical Mechanics
Complex Dynamics in Simple Neural Networks: Understanding Gradient Flow in Phase Retrieval
Despite the widespread use of gradient-based algorithms for optimizing high-dimensional non-convex functions, understanding their ability of finding good minima instead of being trapped in spurious ones remains to a large extent an open problem.
Dynamical mean-field theory for stochastic gradient descent in Gaussian mixture classification
We define a particular stochastic process for which SGD can be extended to a continuous-time limit that we call stochastic gradient flow.
Passed & Spurious: Descent Algorithms and Local Minima in Spiked Matrix-Tensor Models
In this work we analyse quantitatively the interplay between the loss landscape and performance of descent algorithms in a prototypical inference problem, the spiked matrix-tensor model.
Marvels and Pitfalls of the Langevin Algorithm in Noisy High-dimensional Inference
Gradient-descent-based algorithms and their stochastic versions have widespread applications in machine learning and statistical inference.
Approximate Survey Propagation for Statistical Inference
Approximate message passing algorithm enjoyed considerable attention in the last decade.
Glassy nature of the hard phase in inference problems
An algorithmically hard phase was described in a range of inference problems: even if the signal can be reconstructed with a small error from an information theoretic point of view, known algorithms fail unless the noise-to-signal ratio is sufficiently small.
