no code implementations • 9 Sep 2023 • Persia Jana Kamali, Pierfrancesco Urbani
Stochastic Gradient Descent (SGD) is an out-of-equilibrium algorithm used extensively to train artificial neural networks.
no code implementations • 20 Dec 2021 • Francesca Mignacco, Pierfrancesco Urbani
In the under-parametrized regime, where the final training error is positive, the SGD dynamics reaches a stationary state and we define an effective temperature from the fluctuation-dissipation theorem, computed from dynamical mean-field theory.
no code implementations • 8 Mar 2021 • Francesca Mignacco, Pierfrancesco Urbani, Lenka Zdeborová
In this paper we investigate how gradient-based algorithms such as gradient descent, (multi-pass) stochastic gradient descent, its persistent variant, and the Langevin algorithm navigate non-convex loss-landscapes and which of them is able to reach the best generalization error at limited sample complexity.
no code implementations • NeurIPS 2021 • Stefano Sarao Mannelli, Pierfrancesco Urbani
The optimization step in many machine learning problems rarely relies on vanilla gradient descent but it is common practice to use momentum-based accelerated methods.
no code implementations • 4 Jan 2021 • Pierfrancesco Urbani
In the noisy case one has a set of controllable stochastic processes and a cost function that is a functional of their trajectories.
Optimization and Control Disordered Systems and Neural Networks
no code implementations • 21 Dec 2020 • Eran Bouchbinder, Edan Lerner, Corrado Rainone, Pierfrancesco Urbani, Francesco Zamponi
We study a recently introduced and exactly solvable mean-field model for the density of vibrational states $\mathcal{D}(\omega)$ of a structurally disordered system.
Disordered Systems and Neural Networks Soft Condensed Matter Statistical Mechanics
no code implementations • 5 Oct 2020 • Silvio Franz, Antonio Sclocchi, Pierfrancesco Urbani
This algorithm allows to "surf" between isostatic marginally stable configurations and to investigate some properties of such landscape.
Disordered Systems and Neural Networks Statistical Mechanics
no code implementations • NeurIPS 2020 • Stefano Sarao Mannelli, Giulio Biroli, Chiara Cammarota, Florent Krzakala, Pierfrancesco Urbani, Lenka Zdeborová
Despite the widespread use of gradient-based algorithms for optimizing high-dimensional non-convex functions, understanding their ability of finding good minima instead of being trapped in spurious ones remains to a large extent an open problem.
no code implementations • NeurIPS 2020 • Francesca Mignacco, Florent Krzakala, Pierfrancesco Urbani, Lenka Zdeborová
We define a particular stochastic process for which SGD can be extended to a continuous-time limit that we call stochastic gradient flow.
no code implementations • 1 Feb 2019 • Stefano Sarao Mannelli, Florent Krzakala, Pierfrancesco Urbani, Lenka Zdeborová
In this work we analyse quantitatively the interplay between the loss landscape and performance of descent algorithms in a prototypical inference problem, the spiked matrix-tensor model.
no code implementations • 21 Dec 2018 • Stefano Sarao Mannelli, Giulio Biroli, Chiara Cammarota, Florent Krzakala, Pierfrancesco Urbani, Lenka Zdeborová
Gradient-descent-based algorithms and their stochastic versions have widespread applications in machine learning and statistical inference.
no code implementations • 3 Jul 2018 • Fabrizio Antenucci, Florent Krzakala, Pierfrancesco Urbani, Lenka Zdeborová
Approximate message passing algorithm enjoyed considerable attention in the last decade.
no code implementations • 15 May 2018 • Fabrizio Antenucci, Silvio Franz, Pierfrancesco Urbani, Lenka Zdeborová
An algorithmically hard phase was described in a range of inference problems: even if the signal can be reconstructed with a small error from an information theoretic point of view, known algorithms fail unless the noise-to-signal ratio is sufficiently small.