Search Results for author:
Found 9 papers, 7 papers with code
Stochastic Hessian Fittings with Lie Groups
This paper studies the fitting of Hessian or its inverse for stochastic optimizations using a Hessian fitting criterion from the preconditioned stochastic gradient descent (PSGD) method, which is intimately related to many commonly used second order and adaptive gradient optimizers, e. g., BFGS, Gaussian-Newton and natural gradient descent, AdaGrad, etc.
Curvature-Informed SGD via General Purpose Lie-Group Preconditioners
We present a novel approach to accelerate stochastic gradient descent (SGD) by utilizing curvature information obtained from Hessian-vector products or finite differences of parameters and gradients, similar to the BFGS algorithm.
Independent Vector Analysis with Deep Neural Network Source Priors
This paper studies the density priors for independent vector analysis (IVA) with convolutive speech mixture separation as the exemplary application.
A Triangular Network For Density Estimation
We report a triangular neural network implementation of neural autoregressive flow (NAF).
A Multiclass Multiple Instance Learning Method with Exact Likelihood
We study a multiclass multiple instance learning (MIL) problem where the labels only suggest whether any instance of a class exists or does not exist in a training sample or example.
Preconditioner on Matrix Lie Group for SGD
We study two types of preconditioners and preconditioned stochastic gradient descent (SGD) methods in a unified framework.
Online Second Order Methods for Non-Convex Stochastic Optimizations
This paper proposes a family of online second order methods for possibly non-convex stochastic optimizations based on the theory of preconditioned stochastic gradient descent (PSGD), which can be regarded as an enhance stochastic Newton method with the ability to handle gradient noise and non-convexity simultaneously.
Recurrent neural network training with preconditioned stochastic gradient descent
This paper studies the performance of a recently proposed preconditioned stochastic gradient descent (PSGD) algorithm on recurrent neural network (RNN) training.
Preconditioned Stochastic Gradient Descent
When stochastic gradient is used, it can naturally damp the gradient noise to stabilize SGD.
