Search Results for author:
Found 40 papers, 13 papers with code
Acceleration through spectral density estimation
We develop a framework for designing optimal optimization methods in terms of their average-case runtime.
Universal Asymptotic Optimality of Polyak Momentum
We consider the average-case runtime analysis of algorithms for minimizing quadratic objectives.
Stability-Aware Training of Neural Network Interatomic Potentials with Differentiable Boltzmann Estimators
As a general framework applicable across NNIP architectures and systems, StABlE Training is a powerful tool for training stable and accurate NNIPs, particularly in the absence of large reference datasets.
On the Interplay Between Stepsize Tuning and Progressive Sharpening
Recent empirical work has revealed an intriguing property of deep learning models by which the sharpness (largest eigenvalue of the Hessian) increases throughout optimization until it stabilizes around a critical value at which the optimizer operates at the edge of stability, given a fixed stepsize (Cohen et al, 2022).
On Implicit Bias in Overparameterized Bilevel Optimization
Bilevel problems consist of two nested sub-problems, called the outer and inner problems, respectively.
A Novel Stochastic Gradient Descent Algorithm for Learning Principal Subspaces
In this paper, we derive an algorithm that learns a principal subspace from sample entries, can be applied when the approximate subspace is represented by a neural network, and hence can be scaled to datasets with an effectively infinite number of rows and columns.
When is Momentum Extragradient Optimal? A Polynomial-Based Analysis
The extragradient method has gained popularity due to its robust convergence properties for differentiable games.
Second-order regression models exhibit progressive sharpening to the edge of stability
Recent studies of gradient descent with large step sizes have shown that there is often a regime with an initial increase in the largest eigenvalue of the loss Hessian (progressive sharpening), followed by a stabilization of the eigenvalue near the maximum value which allows convergence (edge of stability).
The Curse of Unrolling: Rate of Differentiating Through Optimization
Computing the Jacobian of the solution of an optimization problem is a central problem in machine learning, with applications in hyperparameter optimization, meta-learning, optimization as a layer, and dataset distillation, to name a few.
Only Tails Matter: Average-Case Universality and Robustness in the Convex Regime
The recently developed average-case analysis of optimization methods allows a more fine-grained and representative convergence analysis than usual worst-case results.
Cutting Some Slack for SGD with Adaptive Polyak Stepsizes
We use this insight to develop new variants of the SPS method that are better suited to nonlinear models.
GradMax: Growing Neural Networks using Gradient Information
The architecture and the parameters of neural networks are often optimized independently, which requires costly retraining of the parameters whenever the architecture is modified.
Efficient and Modular Implicit Differentiation
In this paper, we propose automatic implicit differentiation, an efficient and modular approach for implicit differentiation of optimization problems.
Boosting Variational Inference With Locally Adaptive Step-Sizes
Variational Inference makes a trade-off between the capacity of the variational family and the tractability of finding an approximate posterior distribution.
Bridging the Gap Between Adversarial Robustness and Optimization Bias
We demonstrate that the choice of optimizer, neural network architecture, and regularizer significantly affect the adversarial robustness of linear neural networks, providing guarantees without the need for adversarial training.
SGD in the Large: Average-case Analysis, Asymptotics, and Stepsize Criticality
We propose a new framework, inspired by random matrix theory, for analyzing the dynamics of stochastic gradient descent (SGD) when both number of samples and dimensions are large.
Average-case Acceleration for Bilinear Games and Normal Matrices
First, we show that for zero-sum bilinear games the average-case optimal method is the optimal method for the minimization of the Hamiltonian.
Halting Time is Predictable for Large Models: A Universality Property and Average-case Analysis
In fact, the halting time exhibits a universality property: it is independent of the probability distribution.
Stochastic Frank-Wolfe for Constrained Finite-Sum Minimization
We propose a novel Stochastic Frank-Wolfe (a. k. a.
The Geometry of Sign Gradient Descent
Sign-based optimization methods have become popular in machine learning due to their favorable communication cost in distributed optimization and their surprisingly good performance in neural network training.
Average-case Acceleration Through Spectral Density Estimation
We develop a framework for the average-case analysis of random quadratic problems and derive algorithms that are optimal under this analysis.
A Test for Shared Patterns in Cross-modal Brain Activation Analysis
In the last decade, the identification of shared activity patterns has been mostly framed as a supervised learning problem.
The Difficulty of Training Sparse Neural Networks
Additionally, our attempts to find a decreasing objective path from "bad" solutions to the "good" ones in the sparse subspace fail.
On the interplay between noise and curvature and its effect on optimization and generalization
The speed at which one can minimize an expected loss using stochastic methods depends on two properties: the curvature of the loss and the variance of the gradients.
Variance Reduced Three Operator Splitting
This is due to the fact that existing methods require to evaluate the proximity operator for the nonsmooth terms, which can be a costly operation for complex penalties.
Optimization and Control 65K10
Frank-Wolfe Splitting via Augmented Lagrangian Method
In this work, we develop and analyze the Frank-Wolfe Augmented Lagrangian (FW-AL) algorithm, a method for minimizing a smooth function over convex compact sets related by a "linear consistency" constraint that only requires access to a linear minimization oracle over the individual constraints.
Adaptive Three Operator Splitting
We propose and analyze an adaptive step-size variant of the Davis-Yin three operator splitting.
Frank-Wolfe with Subsampling Oracle
The first algorithm that we propose is a randomized variant of the original FW algorithm and achieves a $\mathcal{O}(1/t)$ sublinear convergence rate as in the deterministic counterpart.
Improved asynchronous parallel optimization analysis for stochastic incremental methods
Notably, we prove that ASAGA and KROMAGNON can obtain a theoretical linear speedup on multi-core systems even without sparsity assumptions.
Breaking the Nonsmooth Barrier: A Scalable Parallel Method for Composite Optimization
Due to their simplicity and excellent performance, parallel asynchronous variants of stochastic gradient descent have become popular methods to solve a wide range of large-scale optimization problems on multi-core architectures.
On the convergence rate of the three operator splitting scheme
The three operator splitting scheme was recently proposed by [Davis and Yin, 2015] as a method to optimize composite objective functions with one convex smooth term and two convex (possibly non-smooth) terms for which we have access to their proximity operator.
ASAGA: Asynchronous Parallel SAGA
We describe ASAGA, an asynchronous parallel version of the incremental gradient algorithm SAGA that enjoys fast linear convergence rates.
Hyperparameter optimization with approximate gradient
Most models in machine learning contain at least one hyperparameter to control for model complexity.
Machine Learning for Neuroimaging with Scikit-Learn
Statistical machine learning methods are increasingly used for neuroimaging data analysis.
On the Consistency of Ordinal Regression Methods
We will see that, for a family of surrogate loss functions that subsumes support vector ordinal regression and ORBoosting, consistency can be fully characterized by the derivative of a real-valued function at zero, as happens for convex margin-based surrogates in binary classification.
Data-driven HRF estimation for encoding and decoding models
We develop a method for the joint estimation of activation and HRF using a rank constraint causing the estimated HRF to be equal across events/conditions, yet permitting it to be different across voxels.
API design for machine learning software: experiences from the scikit-learn project
Scikit-learn is an increasingly popular machine learning li- brary.
Second order scattering descriptors predict fMRI activity due to visual textures
Second layer scattering descriptors are known to provide good classification performance on natural quasi-stationary processes such as visual textures due to their sensitivity to higher order moments and continuity with respect to small deformations.
HRF estimation improves sensitivity of fMRI encoding and decoding models
Extracting activation patterns from functional Magnetic Resonance Images (fMRI) datasets remains challenging in rapid-event designs due to the inherent delay of blood oxygen level-dependent (BOLD) signal.
Scikit-learn: Machine Learning in Python
Scikit-learn is a Python module integrating a wide range of state-of-the-art machine learning algorithms for medium-scale supervised and unsupervised problems.
