Search Results for author:
Found 22 papers, 2 papers with code
Second-Order Fine-Tuning without Pain for LLMs:A Hessian Informed Zeroth-Order Optimizer
Fine-tuning large language models (LLMs) with classic first-order optimizers entails prohibitive GPU memory due to the backpropagation process.
PPFL: A Personalized Federated Learning Framework for Heterogeneous Population
Personalization aims to characterize individual preferences and is widely applied across many fields.
Decentralized Riemannian Conjugate Gradient Method on the Stiefel Manifold
This paper proposes a decentralized Riemannian conjugate gradient descent (DRCGD) method that aims at minimizing a global function over the Stiefel manifold.
Mirror Natural Evolution Strategies
In this paper, we focus on the theory of zeroth-order optimization which utilizes both the first-order and second-order information approximated by the zeroth-order queries.
An Efficient Stochastic Algorithm for Decentralized Nonconvex-Strongly-Concave Minimax Optimization
This paper studies the stochastic optimization for decentralized nonconvex-strongly-concave (NC-SC) minimax problems over a multi-agent network.
An Optimal Stochastic Algorithm for Decentralized Nonconvex Finite-sum Optimization
This paper studies the decentralized nonconvex optimization problem $\min_{x\in{\mathbb R}^d} f(x)\triangleq \frac{1}{m}\sum_{i=1}^m f_i(x)$, where $f_i(x)\triangleq \frac{1}{n}\sum_{j=1}^n f_{i, j}(x)$ is the local function on the $i$-th agent of the network.
Decentralized Stochastic Variance Reduced Extragradient Method
This paper studies decentralized convex-concave minimax optimization problems of the form $\min_x\max_y f(x, y) \triangleq\frac{1}{m}\sum_{i=1}^m f_i(x, y)$, where $m$ is the number of agents and each local function can be written as $f_i(x, y)=\frac{1}{n}\sum_{j=1}^n f_{i, j}(x, y)$.
Greedy and Random Quasi-Newton Methods with Faster Explicit Superlinear Convergence
In this paper, we follow Rodomanov and Nesterov’s work to study quasi-Newton methods.
Eigencurve: Optimal Learning Rate Schedule for SGD on Quadratic Objectives with Skewed Hessian Spectrums
In this paper, we propose Eigencurve, the first family of learning rate schedules that can achieve minimax optimal convergence rates (up to a constant) for SGD on quadratic objectives when the eigenvalue distribution of the underlying Hessian matrix is skewed.
DeEPCA: Decentralized Exact PCA with Linear Convergence Rate
This leads to a decentralized PCA algorithm called \texttt{DeEPCA}, which has a convergence rate similar to that of the centralized PCA, while achieving the best communication complexity among existing decentralized PCA algorithms.
PMGT-VR: A decentralized proximal-gradient algorithmic framework with variance reduction
This paper considers the decentralized composite optimization problem.
Decentralized Accelerated Proximal Gradient Descent
In this paper, we propose a new method which establishes the optimal computational complexity and a near optimal communication complexity.
Revisiting Co-Occurring Directions: Sharper Analysis and Efficient Algorithm for Sparse Matrices
We study the streaming model for approximate matrix multiplication (AMM).
Multi-consensus Decentralized Accelerated Gradient Descent
This paper considers the decentralized convex optimization problem, which has a wide range of applications in large-scale machine learning, sensor networks, and control theory.
MiLeNAS: Efficient Neural Architecture Search via Mixed-Level Reformulation
To remedy this, this paper proposes \mldas, a mixed-level reformulation for NAS that can be optimized efficiently and reliably.
Stochastic Recursive Gradient Descent Ascent for Stochastic Nonconvex-Strongly-Concave Minimax Problems
We consider nonconvex-concave minimax optimization problems of the form $\min_{\bf x}\max_{\bf y\in{\mathcal Y}} f({\bf x},{\bf y})$, where $f$ is strongly-concave in $\bf y$ but possibly nonconvex in $\bf x$ and ${\mathcal Y}$ is a convex and compact set.
Fast Generalized Matrix Regression with Applications in Machine Learning
Fast matrix algorithms have become the fundamental tools of machine learning in big data era.
Mirror Natural Evolution Strategies
We show that the estimated covariance matrix of MiNES converges to the inverse of Hessian matrix of the objective function with a sublinear convergence rate.
Hessian-Aware Zeroth-Order Optimization for Black-Box Adversarial Attack
Zeroth-order optimization is an important research topic in machine learning.
Nesterov's Acceleration For Approximate Newton
Besides, the accelerated regularized sub-sampled Newton has good performance comparable to or even better than classical algorithms.
Approximate Newton Methods and Their Local Convergence
We propose a unifying framework to analyze local convergence properties of second order methods.
Nestrov's Acceleration For Second Order Method
Besides, the accelerated regularized sub-sampled Newton has good performance comparable to or even better than state-of-art algorithms.
