Search Results for author:
Found 14 papers, 2 papers with code
Geometric Methods for Sampling, Optimisation, Inference and Adaptive Agents
In this chapter, we identify fundamental geometric structures that underlie the problems of sampling, optimisation, inference and adaptive decision-making.
Optimization on manifolds: A symplectic approach
Optimization tasks are crucial in statistical machine learning.
Implicit Acceleration of Gradient Flow in Overparameterized Linear Models
More precisely, gradient flow preserves the difference of the Gramian~matrices of the input and output weights and we show that the amount of acceleration depends on both the magnitude of that difference (which is fixed at initialization) and the spectrum of the data.
Distributed Optimization, Averaging via ADMM, and Network Topology
For simple algorithms such as gradient descent the dependency of the convergence time with the topology of this network is well-known.
On dissipative symplectic integration with applications to gradient-based optimization
More specifically, we show that a generalization of symplectic integrators to nonconservative and in particular dissipative Hamiltonian systems is able to preserve rates of convergence up to a controlled error.
Gradient flows and proximal splitting methods: A unified view on accelerated and stochastic optimization
We show that similar discretization schemes applied to Newton's equation with an additional dissipative force, which we refer to as accelerated gradient flow, allow us to obtain accelerated variants of all these proximal algorithms -- the majority of which are new although some recover known cases in the literature.
Conformal Symplectic and Relativistic Optimization
Arguably, the two most popular accelerated or momentum-based optimization methods in machine learning are Nesterov's accelerated gradient and Polyaks's heavy ball, both corresponding to different discretizations of a particular second order differential equation with friction.
A Nonsmooth Dynamical Systems Perspective on Accelerated Extensions of ADMM
Recently, there has been great interest in connections between continuous-time dynamical systems and optimization methods, notably in the context of accelerated methods for smooth and unconstrained problems.
An Explicit Convergence Rate for Nesterov's Method from SDP
The framework of Integral Quadratic Constraints (IQC) introduced by Lessard et al. (2014) reduces the computation of upper bounds on the convergence rate of several optimization algorithms to semi-definite programming (SDP).
Kernel k-Groups via Hartigan's Method
In this paper, we consider a formulation for the clustering problem using a weighted version of energy statistics in spaces of negative type.
How is Distributed ADMM Affected by Network Topology?
Here we provide a full characterization of the convergence of distributed over-relaxed ADMM for the same type of consensus problem in terms of the topology of the underlying graph.
Markov Chain Lifting and Distributed ADMM
The time to converge to the steady state of a finite Markov chain can be greatly reduced by a lifting operation, which creates a new Markov chain on an expanded state space.
Tuning Over-Relaxed ADMM
The framework of Integral Quadratic Constraints (IQC) reduces the computation of upper bounds on the convergence rate of several optimization algorithms to a semi-definite program (SDP).
An Explicit Rate Bound for the Over-Relaxed ADMM
In this paper we provide an exact analytical solution to this SDP and obtain a general and explicit upper bound on the convergence rate of the entire family of over-relaxed ADMM.
