Search Results for author:
Found 13 papers, 8 papers with code
Can We Remove the Square-Root in Adaptive Gradient Methods? A Second-Order Perspective
Adaptive gradient optimizers like Adam(W) are the default training algorithms for many deep learning architectures, such as transformers.
Structured Inverse-Free Natural Gradient: Memory-Efficient & Numerically-Stable KFAC for Large Neural Nets
Second-order methods for deep learning -- such as KFAC -- can be useful for neural net training.
Simplifying Momentum-based Positive-definite Submanifold Optimization with Applications to Deep Learning
Riemannian submanifold optimization with momentum is computationally challenging because, to ensure that the iterates remain on the submanifold, we often need to solve difficult differential equations.
Structured second-order methods via natural gradient descent
In this paper, we propose new structured second-order methods and structured adaptive-gradient methods obtained by performing natural-gradient descent on structured parameter spaces.
Tractable structured natural gradient descent using local parameterizations
Natural-gradient descent (NGD) on structured parameter spaces (e. g., low-rank covariances) is computationally challenging due to difficult Fisher-matrix computations.
Handling the Positive-Definite Constraint in the Bayesian Learning Rule
The Bayesian learning rule is a natural-gradient variational inference method, which not only contains many existing learning algorithms as special cases but also enables the design of new algorithms.
Stein's Lemma for the Reparameterization Trick with Exponential Family Mixtures
Our generalization enables us to establish a connection between Stein's lemma and the reparamterization trick to derive gradients of expectations of a large class of functions under weak assumptions.
Fast and Simple Natural-Gradient Variational Inference with Mixture of Exponential-family Approximations
Natural-gradient methods enable fast and simple algorithms for variational inference, but due to computational difficulties, their use is mostly limited to \emph{minimal} exponential-family (EF) approximations.
Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam
Uncertainty computation in deep learning is essential to design robust and reliable systems.
Variational Message Passing with Structured Inference Networks
Recent efforts on combining deep models with probabilistic graphical models are promising in providing flexible models that are also easy to interpret.
Variational Adaptive-Newton Method for Explorative Learning
We present the Variational Adaptive Newton (VAN) method which is a black-box optimization method especially suitable for explorative-learning tasks such as active learning and reinforcement learning.
Conjugate-Computation Variational Inference : Converting Variational Inference in Non-Conjugate Models to Inferences in Conjugate Models
In this paper, we propose a new algorithm called Conjugate-computation Variational Inference (CVI) which brings the best of the two worlds together -- it uses conjugate computations for the conjugate terms and employs stochastic gradients for the rest.
Faster Stochastic Variational Inference using Proximal-Gradient Methods with General Divergence Functions
We also give a convergence-rate analysis of our method and many other previous methods which exploit the geometry of the space.
