Search Results for author:
Found 92 papers, 43 papers with code
SymAgent: A Neural-Symbolic Self-Learning Agent Framework for Complex Reasoning over Knowledge Graphs
However, existing methods face two limitations: 1) they typically assume that all answers to the questions are contained in KGs, neglecting the incompleteness issue of KGs, and 2) they treat the KG as a static repository and overlook the implicit logical reasoning structures inherent in KGs.
Deeply Learned Robust Matrix Completion for Large-scale Low-rank Data Recovery
This paper proposes a novel scalable and learnable non-convex approach, coined Learned Robust Matrix Completion (LRMC), for large-scale RMC problems.
S$^3$Attention: Improving Long Sequence Attention with Smoothed Skeleton Sketching
Attention based models have achieved many remarkable breakthroughs in numerous applications.
Solving General Natural-Language-Description Optimization Problems with Large Language Models
Optimization problems seek to find the best solution to an objective under a set of constraints, and have been widely investigated in real-world applications.
Expressive Power of Graph Neural Networks for (Mixed-Integer) Quadratic Programs
In this work, we investigate the expressive or representative power of GNNs, a crucial aspect of neural network theory, specifically in the context of QP tasks, with both continuous and mixed-integer settings.
ODE-based Learning to Optimize
Then, we formulate a novel learning to optimize (L2O) problem aimed at minimizing the stopping time subject to the convergence and stability condition.
Learning to optimize: A tutorial for continuous and mixed-integer optimization
Learning to Optimize (L2O) stands at the intersection of traditional optimization and machine learning, utilizing the capabilities of machine learning to enhance conventional optimization techniques.
Subsampled Ensemble Can Improve Generalization Tail Exponentially
Ensemble learning is a popular technique to improve the accuracy of machine learning models.
Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark
In the evolving landscape of natural language processing (NLP), fine-tuning pre-trained Large Language Models (LLMs) with first-order (FO) optimizers like SGD and Adam has become standard.
Rethinking the Capacity of Graph Neural Networks for Branching Strategy
In the literature, message-passing GNN (MP-GNN), as the simplest GNN structure, is frequently used as a fast approximation of SB and we find that not all MILPs's SB can be represented with MP-GNN.
DIG-MILP: a Deep Instance Generator for Mixed-Integer Linear Programming with Feasibility Guarantee
Mixed-integer linear programming (MILP) stands as a notable NP-hard problem pivotal to numerous crucial industrial applications.
A Human-on-the-Loop Optimization Autoformalism Approach for Sustainability
This paper outlines a natural conversational approach to solving personalized energy-related problems using large language models (LLMs).
DSGD-CECA: Decentralized SGD with Communication-Optimal Exact Consensus Algorithm
Their communication, governed by the communication topology and gossip weight matrices, facilitates the exchange of model updates.
Towards Constituting Mathematical Structures for Learning to Optimize
Learning to Optimize (L2O), a technique that utilizes machine learning to learn an optimization algorithm automatically from data, has gained arising attention in recent years.
Lower Bounds and Accelerated Algorithms in Distributed Stochastic Optimization with Communication Compression
In this paper, we investigate the performance limit of distributed stochastic optimization algorithms employing communication compression.
An Improved Analysis of (Variance-Reduced) Policy Gradient and Natural Policy Gradient Methods
In this paper, we revisit and improve the convergence of policy gradient (PG), natural PG (NPG) methods, and their variance-reduced variants, under general smooth policy parametrizations.
Alternating Implicit Projected SGD and Its Efficient Variants for Equality-constrained Bilevel Optimization
By leveraging the special structure of the equality constraints problem, the paper first presents an alternating implicit projected SGD approach and establishes the $\tilde{\cal O}(\epsilon^{-2})$ sample complexity that matches the state-of-the-art complexity of ALSET \citep{chen2021closing} for unconstrained bilevel problems.
On Representing Mixed-Integer Linear Programs by Graph Neural Networks
While Mixed-integer linear programming (MILP) is NP-hard in general, practical MILP has received roughly 100--fold speedup in the past twenty years.
Communication-Efficient Topologies for Decentralized Learning with $O(1)$ Consensus Rate
In the proposed family, EquiStatic has a degree of $\Theta(\ln(n))$, where $n$ is the network size, and a series of time-dependent one-peer topologies, EquiDyn, has a constant degree of 1.
On Representing Linear Programs by Graph Neural Networks
In particular, the graph neural network (GNN) is considered a suitable ML model for optimization problems whose variables and constraints are permutation--invariant, for example, the linear program (LP).
Lower Bounds and Nearly Optimal Algorithms in Distributed Learning with Communication Compression
We establish a convergence lower bound for algorithms whether using unbiased or contractive compressors in unidirection or bidirection.
FiLM: Frequency improved Legendre Memory Model for Long-term Time Series Forecasting
Recent studies have shown that deep learning models such as RNNs and Transformers have brought significant performance gains for long-term forecasting of time series because they effectively utilize historical information.
Ranked #3 on
Time Series Forecasting
on ETTh2 (96) Univariate
A Novel Convergence Analysis for Algorithms of the Adam Family
Although rigorous convergence analysis exists for Adam, they impose specific requirements on the update of the adaptive step size, which are not generic enough to cover many other variants of Adam.
Closing the Gap: Tighter Analysis of Alternating Stochastic Gradient Methods for Bilevel Problems
By leveraging the hidden smoothness of the problem, this paper presents a tighter analysis of ALSET for stochastic nested problems.
An Improved Analysis and Rates for Variance Reduction under Without-replacement Sampling Orders
In this paper, we will improve the convergence analysis and rates of variance reduction under without-replacement sampling orders for composite finite-sum minimization. Our results are in two-folds.
BlueFog: Make Decentralized Algorithms Practical for Optimization and Deep Learning
On mainstream DNN training tasks, BlueFog reaches a much higher throughput and achieves an overall $1. 2\times \sim 1. 8\times$ speedup over Horovod, a state-of-the-art distributed deep learning package based on Ring-Allreduce.
Hyperparameter Tuning is All You Need for LISTA
Learned Iterative Shrinkage-Thresholding Algorithm (LISTA) introduces the concept of unrolling an iterative algorithm and training it like a neural network.
Exponential Graph is Provably Efficient for Decentralized Deep Training
Experimental results on a variety of tasks and models demonstrate that decentralized (momentum) SGD over exponential graphs promises both fast and high-quality training.
Learned Robust PCA: A Scalable Deep Unfolding Approach for High-Dimensional Outlier Detection
Robust principal component analysis (RPCA) is a critical tool in modern machine learning, which detects outliers in the task of low-rank matrix reconstruction.
Communicate Then Adapt: An Effective Decentralized Adaptive Method for Deep Training
Decentralized adaptive gradient methods, in which each node averages only with its neighbors, are critical to save communication and wall-clock training time in deep learning tasks.
Curvature-Aware Derivative-Free Optimization
Zeroth-order methods have been gaining popularity due to the demands of large-scale machine learning applications, and the paper focuses on the selection of the step size $\alpha_k$ in these methods.
Tighter Analysis of Alternating Stochastic Gradient Method for Stochastic Nested Problems
By leveraging the hidden smoothness of the problem, this paper presents a tighter analysis of ALSET for stochastic nested problems.
Operator Splitting for Learning to Predict Equilibria in Convex Games
Systems of competing agents can often be modeled as games.
Accelerating Gossip SGD with Periodic Global Averaging
Communication overhead hinders the scalability of large-scale distributed training.
Unified Convergence Analysis for Adaptive Optimization with Moving Average Estimator
First, we show that an increasing or large enough momentum parameter for the first-order moment used in practice is sufficient to ensure the convergence of adaptive algorithms whose adaptive scaling factors of the step size are bounded.
Feasibility-based Fixed Point Networks
This is accomplished using feasibility-based fixed point networks (F-FPNs).
Improved Analysis and Rates for Variance Reduction under Without-replacement Sampling Orders
In the highly data-heterogeneous scenario, Prox-DFinito with optimal cyclic sampling can attain a sample-size-independent convergence rate, which, to our knowledge, is the first result that can match with uniform-iid-sampling with variance reduction.
DecentLaM: Decentralized Momentum SGD for Large-batch Deep Training
Experimental results on a variety of computer vision tasks and models demonstrate that DecentLaM promises both efficient and high-quality training.
JFB: Jacobian-Free Backpropagation for Implicit Networks
Unlike traditional networks, implicit networks solve a fixed point equation to compute inferences.
Learning to Optimize: A Primer and A Benchmark
It automates the design of an optimization method based on its performance on a set of training problems.
Provably Correct Optimization and Exploration with Non-linear Policies
Policy optimization methods remain a powerful workhorse in empirical Reinforcement Learning (RL), with a focus on neural policies that can easily reason over complex and continuous state and/or action spaces.
A Zeroth-Order Block Coordinate Descent Algorithm for Huge-Scale Black-Box Optimization
We consider the zeroth-order optimization problem in the huge-scale setting, where the dimension of the problem is so large that performing even basic vector operations on the decision variables is infeasible.
A Single-Timescale Method for Stochastic Bilevel Optimization
This paper develops a new optimization method for a class of stochastic bilevel problems that we term Single-Timescale stochAstic BiLevEl optimization (STABLE) method.
Moreau Envelope Augmented Lagrangian Method for Nonconvex Optimization with Linear Constraints
We modify ALM to use a Moreau envelope of the augmented Lagrangian and establish its convergence under conditions that are weaker than those in the literature.
Optimization and Control
Learning A Minimax Optimizer: A Pilot Study
We first present Twin L2O, the first dedicated minimax L2O framework consisting of two LSTMs for updating min and max variables, respectively.
CADA: Communication-Adaptive Distributed Adam
This paper proposes an adaptive stochastic gradient descent method for distributed machine learning, which can be viewed as the communication-adaptive counterpart of the celebrated Adam method - justifying its name CADA.
Hybrid Federated Learning: Algorithms and Implementation
To the best of our knowledge, this is the first formulation and algorithm developed for the hybrid FL.
Attentional-Biased Stochastic Gradient Descent
In this paper, we present a simple yet effective provable method (named ABSGD) for addressing the data imbalance or label noise problem in deep learning.
Ranked #3 on
Image Classification
on iWildCam2020-WILDS
A One-bit, Comparison-Based Gradient Estimator
By treating the gradient as an unknown signal to be recovered, we show how one can use tools from one-bit compressed sensing to construct a robust and reliable estimator of the normalized gradient.
Solving Stochastic Compositional Optimization is Nearly as Easy as Solving Stochastic Optimization
In particular, we apply Adam to SCSC, and the exhibited rate of convergence matches that of the original Adam on non-compositional stochastic optimization.
Wasserstein-based Projections with Applications to Inverse Problems
To bridge this gap, we present a new algorithm that takes samples from the manifold of true data as input and outputs an approximation of the projection operator onto this manifold.
An Improved Analysis of Stochastic Gradient Descent with Momentum
Furthermore, the role of dynamic parameters has not been addressed.
Optimization and Control
VAFL: a Method of Vertical Asynchronous Federated Learning
Horizontal Federated learning (FL) handles multi-client data that share the same set of features, and vertical FL trains a better predictor that combine all the features from different clients.
FedPD: A Federated Learning Framework with Optimal Rates and Adaptivity to Non-IID Data
Aiming at designing FL algorithms that are provably fast and require as few assumptions as possible, we propose a new algorithm design strategy from the primal-dual optimization perspective.
Zeroth-Order Regularized Optimization (ZORO): Approximately Sparse Gradients and Adaptive Sampling
We consider the problem of minimizing a high-dimensional objective function, which may include a regularization term, using (possibly noisy) evaluations of the function.
Provably Efficient Exploration for Reinforcement Learning Using Unsupervised Learning
Motivated by the prevailing paradigm of using unsupervised learning for efficient exploration in reinforcement learning (RL) problems [tang2017exploration, bellemare2016unifying], we investigate when this paradigm is provably efficient.
Safeguarded Learned Convex Optimization
Our numerical examples show convergence of Safe-L2O algorithms, even when the provided data is not from the distribution of training data.
LASG: Lazily Aggregated Stochastic Gradients for Communication-Efficient Distributed Learning
The new algorithms adaptively choose between fresh and stale stochastic gradients and have convergence rates comparable to the original SGD.
How Does an Approximate Model Help in Reinforcement Learning?
In particular, we provide an algorithm that uses $\widetilde{O}(N/(1-\gamma)^3/\varepsilon^2)$ samples in a generative model to learn an $\varepsilon$-optimal policy, where $\gamma$ is the discount factor and $N$ is the number of near-optimal actions in the approximate model.
XPipe: Efficient Pipeline Model Parallelism for Multi-GPU DNN Training
It allows the overlapping of the pipelines of multiple micro-batches, including those belonging to different mini-batches.
Universal Safeguarded Learned Convex Optimization with Guaranteed Convergence
Inferences by each network form solution estimates, and networks are trained to optimize these estimates for a particular distribution of data.
ODE Analysis of Stochastic Gradient Methods with Optimism and Anchoring for Minimax Problems and GANs
Despite remarkable empirical success, the training dynamics of generative adversarial networks (GAN), which involves solving a minimax game using stochastic gradients, is still poorly understood.
ODE Analysis of Stochastic Gradient Methods with Optimism and Anchoring for Minimax Problems
Despite remarkable empirical success, the training dynamics of generative adversarial networks (GAN), which involves solving a minimax game using stochastic gradients, is still poorly understood.
Plug-and-Play Methods Provably Converge with Properly Trained Denoisers
Plug-and-play (PnP) is a non-convex framework that integrates modern denoising priors, such as BM3D or deep learning-based denoisers, into ADMM or other proximal algorithms.
ALISTA: Analytic Weights Are As Good As Learned Weights in LISTA
In this work, we propose Analytic LISTA (ALISTA), where the weight matrix in LISTA is computed as the solution to a data-free optimization problem, leaving only the stepsize and threshold parameters to data-driven learning.
A2BCD: Asynchronous Acceleration with Optimal Complexity
In this paper, we propose the Asynchronous Accelerated Nonuniform Randomized Block Coordinate Descent algorithm (A2BCD).
AsyncQVI: Asynchronous-Parallel Q-Value Iteration for Discounted Markov Decision Processes with Near-Optimal Sample Complexity
In this paper, we propose AsyncQVI, an asynchronous-parallel Q-value iteration for discounted Markov decision processes whose transition and reward can only be sampled through a generative model.
Markov Chain Block Coordinate Descent
random and cyclic selections are either infeasible or very expensive.
Acceleration of Primal-Dual Methods by Preconditioning and Simple Subproblem Procedures
They reduce a difficult problem to simple subproblems, so they are easy to implement and have many applications.
Optimization and Control
On Markov Chain Gradient Descent
This paper studies Markov chain gradient descent, a variant of stochastic gradient descent where the random samples are taken on the trajectory of a Markov chain.
Theoretical Linear Convergence of Unfolded ISTA and its Practical Weights and Thresholds
In this work, we study unfolded ISTA (Iterative Shrinkage Thresholding Algorithm) for sparse signal recovery.
LAG: Lazily Aggregated Gradient for Communication-Efficient Distributed Learning
This paper presents a new class of gradient methods for distributed machine learning that adaptively skip the gradient calculations to learn with reduced communication and computation.
Redundancy Techniques for Straggler Mitigation in Distributed Optimization and Learning
Performance of distributed optimization and learning systems is bottlenecked by "straggler" nodes and slow communication links, which significantly delay computation.
Denoising Prior Driven Deep Neural Network for Image Restoration
Then, the iterative process is unfolded into a deep neural network, which is composed of multiple denoisers modules interleaved with back-projection (BP) modules that ensure the observation consistencies.
Run-and-Inspect Method for Nonconvex Optimization and Global Optimality Bounds for R-Local Minimizers
If no sufficient decrease is found, the current point is called an approximate $R$-local minimizer.
Straggler Mitigation in Distributed Optimization Through Data Encoding
Slow running or straggler tasks can significantly reduce computation speed in distributed computation.
First and Second Order Methods for Online Convolutional Dictionary Learning
Convolutional sparse representations are a form of sparse representation with a structured, translation invariant dictionary.
Online Convolutional Dictionary Learning
While a number of different algorithms have recently been proposed for convolutional dictionary learning, this remains an expensive problem.
On the Convergence of Asynchronous Parallel Iteration with Unbounded Delays
Recent years have witnessed the surge of asynchronous parallel (async-parallel) iterative algorithms due to problems involving very large-scale data and a large number of decision variables.
Cyclic Coordinate Update Algorithms for Fixed-Point Problems: Analysis and Applications
To this problem, we apply the coordinate-update algorithms, which update only one or a few components of $x$ at each step.
Optimization and Control Computation 90C06, 90C25, 65K05
A Primer on Coordinate Descent Algorithms
This monograph presents a class of algorithms called coordinate descent algorithms for mathematicians, statisticians, and engineers outside the field of optimization.
On Unbounded Delays in Asynchronous Parallel Fixed-Point Algorithms
Existing analysis of ARock assumes the delays to be bounded, and uses this bound to set a step size that is important to both convergence and efficiency.
Coordinate Friendly Structures, Algorithms and Applications
To derive simple subproblems for several new classes of applications, this paper systematically studies coordinate-friendly operators that perform low-cost coordinate updates.
Optimal Sparse Kernel Learning for Hyperspectral Anomaly Detection
In this paper, a novel framework of sparse kernel learning for Support Vector Data Description (SVDD) based anomaly detection is presented.
ARock: an Algorithmic Framework for Asynchronous Parallel Coordinate Updates
The agents share $x$ through either global memory or communication.
A fast patch-dictionary method for whole image recovery
With very few exceptions, this issue has limited the applications of image-patch methods to the local kind of tasks such as denoising, inpainting, cartoon-texture decomposition, super-resolution, and image deblurring, for which one can process a few patches at a time.
Block stochastic gradient iteration for convex and nonconvex optimization
Its convergence for both convex and nonconvex cases are established in different senses.
Sparse Recovery via Differential Inclusions
In this paper, we recover sparse signals from their noisy linear measurements by solving nonlinear differential inclusions, which is based on the notion of inverse scale space (ISS) developed in applied mathematics.
EXTRA: An Exact First-Order Algorithm for Decentralized Consensus Optimization
In this paper, we develop a decentralized algorithm for the consensus optimization problem $$\min\limits_{x\in\mathbb{R}^p}~\bar{f}(x)=\frac{1}{n}\sum\limits_{i=1}^n f_i(x),$$ which is defined over a connected network of $n$ agents, where each function $f_i$ is held privately by agent $i$ and encodes the agent's data and objective.
Optimization and Control
Video Compressive Sensing for Dynamic MRI
We apply this framework to accelerate the acquisition process of dynamic MRI and show it achieves the best reconstruction accuracy with the least computational time compared with existing algorithms in the literature.
Parallel matrix factorization for low-rank tensor completion
Phase transition plots reveal that our algorithm can recover a variety of synthetic low-rank tensors from significantly fewer samples than the compared methods, which include a matrix completion method applied to tensor recovery and two state-of-the-art tensor completion methods.
Numerical Analysis Numerical Analysis Computation
An Alternating Direction Algorithm for Matrix Completion with Nonnegative Factors
By taking the advantages of both nonnegativity and low-rankness, one can generally obtain superior results than those of just using one of the two properties.
Information Theory Numerical Analysis Information Theory Numerical Analysis
