Search Results for author:
Found 9 papers, 3 papers with code
Differentiating Nonsmooth Solutions to Parametric Monotone Inclusion Problems
We leverage path differentiability and a recent result on nonsmooth implicit differentiation calculus to give sufficient conditions ensuring that the solution to a monotone inclusion problem will be path differentiable, with formulas for computing its generalized gradient.
On the complexity of nonsmooth automatic differentiation
Using the notion of conservative gradient, we provide a simple model to estimate the computational costs of the backward and forward modes of algorithmic differentiation for a wide class of nonsmooth programs.
Automatic differentiation of nonsmooth iterative algorithms
Is there a limiting object for nonsmooth piggyback automatic differentiation (AD)?
Numerical influence of ReLU'(0) on backpropagation
In theory, the choice of ReLU(0) in [0, 1] for a neural network has a negligible influence both on backpropagation and training.
Nonsmooth Implicit Differentiation for Machine Learning and Optimization
In view of training increasingly complex learning architectures, we establish a nonsmooth implicit function theorem with an operational calculus.
Second-order step-size tuning of SGD for non-convex optimization
In view of a direct and simple improvement of vanilla SGD, this paper presents a fine-tuning of its step-sizes in the mini-batch case.
A Hölderian backtracking method for min-max and min-min problems
We present a new algorithm to solve min-max or min-min problems out of the convex world.
Conservative set valued fields, automatic differentiation, stochastic gradient method and deep learning
Modern problems in AI or in numerical analysis require nonsmooth approaches with a flexible calculus.
An Inertial Newton Algorithm for Deep Learning
We prove the convergence of INNA for most deep learning problems.
