Search Results for author:
Found 84 papers, 38 papers with code
An Alternating Direction Algorithm for Matrix Completion with Nonnegative Factors
By taking the advantages of both nonnegativity and low-rankness, one can generally obtain superior results than those of just using one of the two properties.
Information Theory Numerical Analysis Information Theory Numerical Analysis
Parallel matrix factorization for low-rank tensor completion
Phase transition plots reveal that our algorithm can recover a variety of synthetic low-rank tensors from significantly fewer samples than the compared methods, which include a matrix completion method applied to tensor recovery and two state-of-the-art tensor completion methods.
Numerical Analysis Numerical Analysis Computation
Video Compressive Sensing for Dynamic MRI
We apply this framework to accelerate the acquisition process of dynamic MRI and show it achieves the best reconstruction accuracy with the least computational time compared with existing algorithms in the literature.
EXTRA: An Exact First-Order Algorithm for Decentralized Consensus Optimization
In this paper, we develop a decentralized algorithm for the consensus optimization problem $$\min\limits_{x\in\mathbb{R}^p}~\bar{f}(x)=\frac{1}{n}\sum\limits_{i=1}^n f_i(x),$$ which is defined over a connected network of $n$ agents, where each function $f_i$ is held privately by agent $i$ and encodes the agent's data and objective.
Optimization and Control
Sparse Recovery via Differential Inclusions
In this paper, we recover sparse signals from their noisy linear measurements by solving nonlinear differential inclusions, which is based on the notion of inverse scale space (ISS) developed in applied mathematics.
Block stochastic gradient iteration for convex and nonconvex optimization
Its convergence for both convex and nonconvex cases are established in different senses.
A fast patch-dictionary method for whole image recovery
With very few exceptions, this issue has limited the applications of image-patch methods to the local kind of tasks such as denoising, inpainting, cartoon-texture decomposition, super-resolution, and image deblurring, for which one can process a few patches at a time.
Optimal Sparse Kernel Learning for Hyperspectral Anomaly Detection
In this paper, a novel framework of sparse kernel learning for Support Vector Data Description (SVDD) based anomaly detection is presented.
ARock: an Algorithmic Framework for Asynchronous Parallel Coordinate Updates
The agents share $x$ through either global memory or communication.
Coordinate Friendly Structures, Algorithms and Applications
To derive simple subproblems for several new classes of applications, this paper systematically studies coordinate-friendly operators that perform low-cost coordinate updates.
On Unbounded Delays in Asynchronous Parallel Fixed-Point Algorithms
Existing analysis of ARock assumes the delays to be bounded, and uses this bound to set a step size that is important to both convergence and efficiency.
A Primer on Coordinate Descent Algorithms
This monograph presents a class of algorithms called coordinate descent algorithms for mathematicians, statisticians, and engineers outside the field of optimization.
Cyclic Coordinate Update Algorithms for Fixed-Point Problems: Analysis and Applications
To this problem, we apply the coordinate-update algorithms, which update only one or a few components of $x$ at each step.
Optimization and Control Computation 90C06, 90C25, 65K05
On the Convergence of Asynchronous Parallel Iteration with Unbounded Delays
Recent years have witnessed the surge of asynchronous parallel (async-parallel) iterative algorithms due to problems involving very large-scale data and a large number of decision variables.
Online Convolutional Dictionary Learning
While a number of different algorithms have recently been proposed for convolutional dictionary learning, this remains an expensive problem.
First and Second Order Methods for Online Convolutional Dictionary Learning
Convolutional sparse representations are a form of sparse representation with a structured, translation invariant dictionary.
Straggler Mitigation in Distributed Optimization Through Data Encoding
Slow running or straggler tasks can significantly reduce computation speed in distributed computation.
Run-and-Inspect Method for Nonconvex Optimization and Global Optimality Bounds for R-Local Minimizers
If no sufficient decrease is found, the current point is called an approximate $R$-local minimizer.
Denoising Prior Driven Deep Neural Network for Image Restoration
Then, the iterative process is unfolded into a deep neural network, which is composed of multiple denoisers modules interleaved with back-projection (BP) modules that ensure the observation consistencies.
Redundancy Techniques for Straggler Mitigation in Distributed Optimization and Learning
Performance of distributed optimization and learning systems is bottlenecked by "straggler" nodes and slow communication links, which significantly delay computation.
LAG: Lazily Aggregated Gradient for Communication-Efficient Distributed Learning
This paper presents a new class of gradient methods for distributed machine learning that adaptively skip the gradient calculations to learn with reduced communication and computation.
Theoretical Linear Convergence of Unfolded ISTA and its Practical Weights and Thresholds
In this work, we study unfolded ISTA (Iterative Shrinkage Thresholding Algorithm) for sparse signal recovery.
On Markov Chain Gradient Descent
This paper studies Markov chain gradient descent, a variant of stochastic gradient descent where the random samples are taken on the trajectory of a Markov chain.
Acceleration of Primal-Dual Methods by Preconditioning and Simple Subproblem Procedures
They reduce a difficult problem to simple subproblems, so they are easy to implement and have many applications.
Optimization and Control
Markov Chain Block Coordinate Descent
random and cyclic selections are either infeasible or very expensive.
AsyncQVI: Asynchronous-Parallel Q-Value Iteration for Discounted Markov Decision Processes with Near-Optimal Sample Complexity
In this paper, we propose AsyncQVI, an asynchronous-parallel Q-value iteration for discounted Markov decision processes whose transition and reward can only be sampled through a generative model.
ALISTA: Analytic Weights Are As Good As Learned Weights in LISTA
In this work, we propose Analytic LISTA (ALISTA), where the weight matrix in LISTA is computed as the solution to a data-free optimization problem, leaving only the stepsize and threshold parameters to data-driven learning.
A2BCD: Asynchronous Acceleration with Optimal Complexity
In this paper, we propose the Asynchronous Accelerated Nonuniform Randomized Block Coordinate Descent algorithm (A2BCD).
Plug-and-Play Methods Provably Converge with Properly Trained Denoisers
Plug-and-play (PnP) is a non-convex framework that integrates modern denoising priors, such as BM3D or deep learning-based denoisers, into ADMM or other proximal algorithms.
ODE Analysis of Stochastic Gradient Methods with Optimism and Anchoring for Minimax Problems
Despite remarkable empirical success, the training dynamics of generative adversarial networks (GAN), which involves solving a minimax game using stochastic gradients, is still poorly understood.
ODE Analysis of Stochastic Gradient Methods with Optimism and Anchoring for Minimax Problems and GANs
Despite remarkable empirical success, the training dynamics of generative adversarial networks (GAN), which involves solving a minimax game using stochastic gradients, is still poorly understood.
Universal Safeguarded Learned Convex Optimization with Guaranteed Convergence
Inferences by each network form solution estimates, and networks are trained to optimize these estimates for a particular distribution of data.
XPipe: Efficient Pipeline Model Parallelism for Multi-GPU DNN Training
It allows the overlapping of the pipelines of multiple micro-batches, including those belonging to different mini-batches.
How Does an Approximate Model Help in Reinforcement Learning?
In particular, we provide an algorithm that uses $\widetilde{O}(N/(1-\gamma)^3/\varepsilon^2)$ samples in a generative model to learn an $\varepsilon$-optimal policy, where $\gamma$ is the discount factor and $N$ is the number of near-optimal actions in the approximate model.
LASG: Lazily Aggregated Stochastic Gradients for Communication-Efficient Distributed Learning
The new algorithms adaptively choose between fresh and stale stochastic gradients and have convergence rates comparable to the original SGD.
Safeguarded Learned Convex Optimization
Our numerical examples show convergence of Safe-L2O algorithms, even when the provided data is not from the distribution of training data.
Provably Efficient Exploration for Reinforcement Learning Using Unsupervised Learning
Motivated by the prevailing paradigm of using unsupervised learning for efficient exploration in reinforcement learning (RL) problems [tang2017exploration, bellemare2016unifying], we investigate when this paradigm is provably efficient.
Zeroth-Order Regularized Optimization (ZORO): Approximately Sparse Gradients and Adaptive Sampling
We consider the problem of minimizing a high-dimensional objective function, which may include a regularization term, using (possibly noisy) evaluations of the function.
FedPD: A Federated Learning Framework with Optimal Rates and Adaptivity to Non-IID Data
Aiming at designing FL algorithms that are provably fast and require as few assumptions as possible, we propose a new algorithm design strategy from the primal-dual optimization perspective.
VAFL: a Method of Vertical Asynchronous Federated Learning
Horizontal Federated learning (FL) handles multi-client data that share the same set of features, and vertical FL trains a better predictor that combine all the features from different clients.
An Improved Analysis of Stochastic Gradient Descent with Momentum
Furthermore, the role of dynamic parameters has not been addressed.
Optimization and Control
Wasserstein-based Projections with Applications to Inverse Problems
To bridge this gap, we present a new algorithm that takes samples from the manifold of true data as input and outputs an approximation of the projection operator onto this manifold.
Solving Stochastic Compositional Optimization is Nearly as Easy as Solving Stochastic Optimization
In particular, we apply Adam to SCSC, and the exhibited rate of convergence matches that of the original Adam on non-compositional stochastic optimization.
A One-bit, Comparison-Based Gradient Estimator
By treating the gradient as an unknown signal to be recovered, we show how one can use tools from one-bit compressed sensing to construct a robust and reliable estimator of the normalized gradient.
Attentional-Biased Stochastic Gradient Descent
In this paper, we present a simple yet effective provable method (named ABSGD) for addressing the data imbalance or label noise problem in deep learning.
Hybrid Federated Learning: Algorithms and Implementation
To the best of our knowledge, this is the first formulation and algorithm developed for the hybrid FL.
CADA: Communication-Adaptive Distributed Adam
This paper proposes an adaptive stochastic gradient descent method for distributed machine learning, which can be viewed as the communication-adaptive counterpart of the celebrated Adam method - justifying its name CADA.
Learning A Minimax Optimizer: A Pilot Study
We first present Twin L2O, the first dedicated minimax L2O framework consisting of two LSTMs for updating min and max variables, respectively.
Moreau Envelope Augmented Lagrangian Method for Nonconvex Optimization with Linear Constraints
We modify ALM to use a Moreau envelope of the augmented Lagrangian and establish its convergence under conditions that are weaker than those in the literature.
Optimization and Control
A Single-Timescale Method for Stochastic Bilevel Optimization
This paper develops a new optimization method for a class of stochastic bilevel problems that we term Single-Timescale stochAstic BiLevEl optimization (STABLE) method.
A Zeroth-Order Block Coordinate Descent Algorithm for Huge-Scale Black-Box Optimization
We consider the zeroth-order optimization problem in the huge-scale setting, where the dimension of the problem is so large that performing even basic vector operations on the decision variables is infeasible.
Provably Correct Optimization and Exploration with Non-linear Policies
Policy optimization methods remain a powerful workhorse in empirical Reinforcement Learning (RL), with a focus on neural policies that can easily reason over complex and continuous state and/or action spaces.
Learning to Optimize: A Primer and A Benchmark
It automates the design of an optimization method based on its performance on a set of training problems.
JFB: Jacobian-Free Backpropagation for Implicit Networks
Unlike traditional networks, implicit networks solve a fixed point equation to compute inferences.
DecentLaM: Decentralized Momentum SGD for Large-batch Deep Training
Experimental results on a variety of computer vision tasks and models demonstrate that DecentLaM promises both efficient and high-quality training.
Improved Analysis and Rates for Variance Reduction under Without-replacement Sampling Orders
In the highly data-heterogeneous scenario, Prox-DFinito with optimal cyclic sampling can attain a sample-size-independent convergence rate, which, to our knowledge, is the first result that can match with uniform-iid-sampling with variance reduction.
Feasibility-based Fixed Point Networks
This is accomplished using feasibility-based fixed point networks (F-FPNs).
A Novel Convergence Analysis for Algorithms of the Adam Family and Beyond
Our analysis exhibits that an increasing or large enough "momentum" parameter for the first-order moment used in practice is sufficient to ensure Adam and its many variants converge under a mild boundness condition on the adaptive scaling factor of the step size.
Accelerating Gossip SGD with Periodic Global Averaging
Communication overhead hinders the scalability of large-scale distributed training.
Operator Splitting for Learning to Predict Equilibria in Convex Games
Systems of competing agents can often be modeled as games.
Tighter Analysis of Alternating Stochastic Gradient Method for Stochastic Nested Problems
By leveraging the hidden smoothness of the problem, this paper presents a tighter analysis of ALSET for stochastic nested problems.
Curvature-Aware Derivative-Free Optimization
Zeroth-order methods have been gaining popularity due to the demands of large-scale machine learning applications, and the paper focuses on the selection of the step size $\alpha_k$ in these methods.
Communicate Then Adapt: An Effective Decentralized Adaptive Method for Deep Training
Decentralized adaptive gradient methods, in which each node averages only with its neighbors, are critical to save communication and wall-clock training time in deep learning tasks.
Learned Robust PCA: A Scalable Deep Unfolding Approach for High-Dimensional Outlier Detection
Robust principal component analysis (RPCA) is a critical tool in modern machine learning, which detects outliers in the task of low-rank matrix reconstruction.
Exponential Graph is Provably Efficient for Decentralized Deep Training
Experimental results on a variety of tasks and models demonstrate that decentralized (momentum) SGD over exponential graphs promises both fast and high-quality training.
Hyperparameter Tuning is All You Need for LISTA
Learned Iterative Shrinkage-Thresholding Algorithm (LISTA) introduces the concept of unrolling an iterative algorithm and training it like a neural network.
BlueFog: Make Decentralized Algorithms Practical for Optimization and Deep Learning
On mainstream DNN training tasks, BlueFog reaches a much higher throughput and achieves an overall $1. 2\times \sim 1. 8\times$ speedup over Horovod, a state-of-the-art distributed deep learning package based on Ring-Allreduce.
An Improved Analysis and Rates for Variance Reduction under Without-replacement Sampling Orders
In this paper, we will improve the convergence analysis and rates of variance reduction under without-replacement sampling orders for composite finite-sum minimization. Our results are in two-folds.
Closing the Gap: Tighter Analysis of Alternating Stochastic Gradient Methods for Bilevel Problems
By leveraging the hidden smoothness of the problem, this paper presents a tighter analysis of ALSET for stochastic nested problems.
A Novel Convergence Analysis for Algorithms of the Adam Family
Although rigorous convergence analysis exists for Adam, they impose specific requirements on the update of the adaptive step size, which are not generic enough to cover many other variants of Adam.
FiLM: Frequency improved Legendre Memory Model for Long-term Time Series Forecasting
Recent studies have shown that deep learning models such as RNNs and Transformers have brought significant performance gains for long-term forecasting of time series because they effectively utilize historical information.
Ranked #3 on
Time Series Forecasting
on ETTh2 (96) Univariate
Lower Bounds and Nearly Optimal Algorithms in Distributed Learning with Communication Compression
We establish a convergence lower bound for algorithms whether using unbiased or contractive compressors in unidirection or bidirection.
On Representing Linear Programs by Graph Neural Networks
In particular, the graph neural network (GNN) is considered a suitable ML model for optimization problems whose variables and constraints are permutation--invariant, for example, the linear program (LP).
Communication-Efficient Topologies for Decentralized Learning with $O(1)$ Consensus Rate
In the proposed family, EquiStatic has a degree of $\Theta(\ln(n))$, where $n$ is the network size, and a series of time-dependent one-peer topologies, EquiDyn, has a constant degree of 1.
On Representing Mixed-Integer Linear Programs by Graph Neural Networks
While Mixed-integer linear programming (MILP) is NP-hard in general, practical MILP has received roughly 100--fold speedup in the past twenty years.
Alternating Implicit Projected SGD and Its Efficient Variants for Equality-constrained Bilevel Optimization
By leveraging the special structure of the equality constraints problem, the paper first presents an alternating implicit projected SGD approach and establishes the $\tilde{\cal O}(\epsilon^{-2})$ sample complexity that matches the state-of-the-art complexity of ALSET \citep{chen2021closing} for unconstrained bilevel problems.
An Improved Analysis of (Variance-Reduced) Policy Gradient and Natural Policy Gradient Methods
In this paper, we revisit and improve the convergence of policy gradient (PG), natural PG (NPG) methods, and their variance-reduced variants, under general smooth policy parametrizations.
Lower Bounds and Accelerated Algorithms in Distributed Stochastic Optimization with Communication Compression
In this paper, we investigate the performance limit of distributed stochastic optimization algorithms employing communication compression.
Towards Constituting Mathematical Structures for Learning to Optimize
Learning to Optimize (L2O), a technique that utilizes machine learning to learn an optimization algorithm automatically from data, has gained arising attention in recent years.
DSGD-CECA: Decentralized SGD with Communication-Optimal Exact Consensus Algorithm
Their communication, governed by the communication topology and gossip weight matrices, facilitates the exchange of model updates.
A Human-on-the-Loop Optimization Autoformalism Approach for Sustainability
This paper outlines a natural conversational approach to solving personalized energy-related problems using large language models (LLMs).
DIG-MILP: a Deep Instance Generator for Mixed-Integer Linear Programming with Feasibility Guarantee
Mixed-integer linear programming (MILP) stands as a notable NP-hard problem pivotal to numerous crucial industrial applications.
Rethinking the Capacity of Graph Neural Networks for Branching Strategy
Graph neural networks (GNNs) have been widely used to predict properties and heuristics of mixed-integer linear programs (MILPs) and hence accelerate MILP solvers.
Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark
In the evolving landscape of natural language processing (NLP), fine-tuning pre-trained Large Language Models (LLMs) with first-order (FO) optimizers like SGD and Adam has become standard.
