Search Results for author:
Found 27 papers, 6 papers with code
A second-order-like optimizer with adaptive gradient scaling for deep learning
In this empirical article, we introduce INNAprop, an optimization algorithm that combines the INNA method with the RMSprop adaptive gradient scaling.
Derivatives of Stochastic Gradient Descent in parametric optimization
We investigate the behavior of the derivatives of the iterates of Stochastic Gradient Descent (SGD) with respect to that parameter and show that they are driven by an inexact SGD recursion on a different objective function, perturbed by the convergence of the original SGD.
Inexact subgradient methods for semialgebraic functions
Motivated by the widespread use of approximate derivatives in machine learning and optimization, we study inexact subgradient methods with non-vanishing additive errors and step sizes.
Differentiating Nonsmooth Solutions to Parametric Monotone Inclusion Problems
We leverage path differentiability and a recent result on nonsmooth implicit differentiation calculus to give sufficient conditions ensuring that the solution to a monotone inclusion problem will be path differentiable, with formulas for computing its generalized gradient.
The derivatives of Sinkhorn-Knopp converge
We show that the derivatives of the Sinkhorn-Knopp algorithm, or iterative proportional fitting procedure, converge towards the derivatives of the entropic regularization of the optimal transport problem with a locally uniform linear convergence rate.
On the complexity of nonsmooth automatic differentiation
Using the notion of conservative gradient, we provide a simple model to estimate the computational costs of the backward and forward modes of algorithmic differentiation for a wide class of nonsmooth programs.
Automatic differentiation of nonsmooth iterative algorithms
Is there a limiting object for nonsmooth piggyback automatic differentiation (AD)?
Path differentiability of ODE flows
We consider flows of ordinary differential equations (ODEs) driven by path differentiable vector fields.
Numerical influence of ReLU'(0) on backpropagation
In theory, the choice of ReLU(0) in [0, 1] for a neural network has a negligible influence both on backpropagation and training.
Nonsmooth Implicit Differentiation for Machine Learning and Optimization
In view of training increasingly complex learning architectures, we establish a nonsmooth implicit function theorem with an operational calculus.
Nonsmooth Implicit Differentiation for Machine-Learning and Optimization
In view of training increasingly complex learning architectures, we establish a nonsmooth implicit function theorem with an operational calculus.
Numerical influence of ReLU’(0) on backpropagation
In theory, the choice of ReLU(0) in [0, 1] for a neural network has a negligible influence both on backpropagation and training.
Second-order step-size tuning of SGD for non-convex optimization
In view of a direct and simple improvement of vanilla SGD, this paper presents a fine-tuning of its step-sizes in the mini-batch case.
A Sublevel Moment-SOS Hierarchy for Polynomial Optimization
We introduce a sublevel Moment-SOS hierarchy where each SDP relaxation can be viewed as an intermediate (or interpolation) between the d-th and (d+1)-th order SDP relaxations of the Moment-SOS hierarchy (dense or sparse version).
Combinatorial Optimization
Optimization and Control
Sequential convergence of AdaGrad algorithm for smooth convex optimization
We prove that the iterates produced by, either the scalar step size variant, or the coordinatewise variant of AdaGrad algorithm, are convergent sequences when applied to convex objective functions with Lipschitz gradient.
A Hölderian backtracking method for min-max and min-min problems
We present a new algorithm to solve min-max or min-min problems out of the convex world.
Incremental Without Replacement Sampling in Nonconvex Optimization
Minibatch decomposition methods for empirical risk minimization are commonly analysed in a stochastic approximation setting, also known as sampling with replacement.
A mathematical model for automatic differentiation in machine learning
Automatic differentiation, as implemented today, does not have a simple mathematical model adapted to the needs of modern machine learning.
Semialgebraic Optimization for Lipschitz Constants of ReLU Networks
The Lipschitz constant of a network plays an important role in many applications of deep learning, such as robustness certification and Wasserstein Generative Adversarial Network.
Rate of convergence for geometric inference based on the empirical Christoffel function
We consider the problem of estimating the support of a measure from a finite, independent, sample.
Fairness with Wasserstein Adversarial Networks
For both models, we devise a learning algorithm based on approximation of Wasserstein distances using adversarial networks.
Conservative set valued fields, automatic differentiation, stochastic gradient method and deep learning
Modern problems in AI or in numerical analysis require nonsmooth approaches with a flexible calculus.
An Inertial Newton Algorithm for Deep Learning
We prove the convergence of INNA for most deep learning problems.
Data analysis from empirical moments and the Christoffel function
Spectral features of the empirical moment matrix constitute a resourceful tool for unveiling properties of a cloud of points, among which, density, support and latent structures.
Relating Leverage Scores and Density using Regularized Christoffel Functions
Statistical leverage scores emerged as a fundamental tool for matrix sketching and column sampling with applications to low rank approximation, regression, random feature learning and quadrature.
The empirical Christoffel function with applications in data analysis
Secondly, we provide a consistency result which relates the empirical Christoffel function and its population counterpart in the limit of large samples.
Sorting out typicality with the inverse moment matrix SOS polynomial
In fact, this SOS polynomial is directly related to orthogonal polynomials and the Christoffel function.
