Search Results for author:
Found 16 papers, 3 papers with code
Second-order Information Promotes Mini-Batch Robustness in Variance-Reduced Gradients
We show that, for finite-sum minimization problems, incorporating partial second-order information of the objective function can dramatically improve the robustness to mini-batch size of variance-reduced stochastic gradient methods, making them more scalable while retaining their benefits over traditional Newton-type approaches.
Non-Uniform Smoothness for Gradient Descent
The analysis of gradient descent-type methods typically relies on the Lipschitz continuity of the objective gradient.
Adaptive Consensus: A network pruning approach for decentralized optimization
We consider network-based decentralized optimization problems, where each node in the network possesses a local function and the objective is to collectively attain a consensus solution that minimizes the sum of all the local functions.
Collaborative and Distributed Bayesian Optimization via Consensus: Showcasing the Power of Collaboration for Optimal Design
Empirically, through simulated datasets and a real-world collaborative sensor design experiment, we show that our framework can effectively accelerate and improve the optimal design process and benefit all participants.
A Sequential Quadratic Programming Method with High Probability Complexity Bounds for Nonlinear Equality Constrained Stochastic Optimization
A step-search sequential quadratic programming method is proposed for solving nonlinear equality constrained stochastic optimization problems.
A Stochastic Sequential Quadratic Optimization Algorithm for Nonlinear Equality Constrained Optimization with Rank-Deficient Jacobians
A sequential quadratic optimization algorithm is proposed for solving smooth nonlinear equality constrained optimization problems in which the objective function is defined by an expectation of a stochastic function.
SONIA: A Symmetric Blockwise Truncated Optimization Algorithm
This work presents a new algorithm for empirical risk minimization.
Finite Difference Neural Networks: Fast Prediction of Partial Differential Equations
Discovering the underlying behavior of complex systems is an important topic in many science and engineering disciplines.
Scaling Up Quasi-Newton Algorithms: Communication Efficient Distributed SR1
In this paper, we present a scalable distributed implementation of the Sampled Limited-memory Symmetric Rank-1 (S-LSR1) algorithm.
Linear interpolation gives better gradients than Gaussian smoothing in derivative-free optimization
We then demonstrate via rigorous analysis of the variance and by numerical comparisons on reinforcement learning tasks that the Gaussian sampling method used in [Salimans et al. 2016] is significantly inferior to the orthogonal sampling used in [Choromaski et al. 2018] as well as more general interpolation methods.
A Theoretical and Empirical Comparison of Gradient Approximations in Derivative-Free Optimization
To this end, we use the results in [Berahas et al., 2019] and show how each method can satisfy the sufficient conditions, possibly only with some sufficiently large probability at each iteration, as happens to be the case with Gaussian smoothing and smoothing on a sphere.
Optimization and Control
Quasi-Newton Methods for Machine Learning: Forget the Past, Just Sample
We present two sampled quasi-Newton methods (sampled LBFGS and sampled LSR1) for solving empirical risk minimization problems that arise in machine learning.
A Robust Multi-Batch L-BFGS Method for Machine Learning
This paper describes an implementation of the L-BFGS method designed to deal with two adversarial situations.
An Investigation of Newton-Sketch and Subsampled Newton Methods
Sketching, a dimensionality reduction technique, has received much attention in the statistics community.
A Multi-Batch L-BFGS Method for Machine Learning
The question of how to parallelize the stochastic gradient descent (SGD) method has received much attention in the literature.
adaQN: An Adaptive Quasi-Newton Algorithm for Training RNNs
In this paper, we present adaQN, a stochastic quasi-Newton algorithm for training RNNs.
