Search Results for author:
Found 36 papers, 7 papers with code
Beyond Single-Model Views for Deep Learning: Optimization versus Generalizability of Stochastic Optimization Algorithms
Our investigation encompasses a wide array of techniques, including SGD and its variants, flat-minima optimizers, and new algorithms we propose under the Basin Hopping framework.
Bilevel Optimization under Unbounded Smoothness: A New Algorithm and Convergence Analysis
When the upper-level problem is nonconvex and unbounded smooth, and the lower-level problem is strongly convex, we prove that our algorithm requires $\widetilde{\mathcal{O}}(1/\epsilon^4)$ iterations to find an $\epsilon$-stationary point in the stochastic setting, where each iteration involves calling a stochastic gradient or Hessian-vector product oracle.
Stability and Generalization for Minibatch SGD and Local SGD
We show both minibatch and local SGD achieve a linear speedup to attain the optimal risk bounds.
EPISODE: Episodic Gradient Clipping with Periodic Resampled Corrections for Federated Learning with Heterogeneous Data
In this paper, we design EPISODE, the very first algorithm to solve FL problems with heterogeneous data in the nonconvex and relaxed smoothness setting.
On Inferring User Socioeconomic Status with Mobility Records
The DeepSEI model incorporates two networks called deep network and recurrent network, which extract the features of the mobility records from three aspects, namely spatiality, temporality and activity, one at a coarse level and the other at a detailed level.
Robustness to Unbounded Smoothness of Generalized SignSGD
We also compare these algorithms with popular optimizers on a set of deep learning tasks, observing that we can match the performance of Adam while beating the others.
Fast Composite Optimization and Statistical Recovery in Federated Learning
This paper investigates a class of composite optimization and statistical recovery problems in the FL setting, whose loss function consists of a data-dependent smooth loss and a non-smooth regularizer.
Will Bilevel Optimizers Benefit from Loops
Existing studies in the literature cover only some of those implementation choices, and the complexity bounds available are not refined enough to enable rigorous comparison among different implementations.
A Communication-Efficient Distributed Gradient Clipping Algorithm for Training Deep Neural Networks
Gradient clipping is usually employed to address this issue in the single machine setting, but exploring this technique in the distributed setting is still in its infancy: it remains mysterious whether the gradient clipping scheme can take advantage of multiple machines to enjoy parallel speedup.
Understanding AdamW through Proximal Methods and Scale-Freeness
First, we show how to re-interpret AdamW as an approximation of a proximal gradient method, which takes advantage of the closed-form proximal mapping of the regularizer instead of only utilizing its gradient information as in Adam-$\ell_2$.
Loss Landscape Dependent Self-Adjusting Learning Rates in Decentralized Stochastic Gradient Descent
We conduct extensive studies over 18 state-of-the-art DL models/tasks and demonstrate that DPSGD often converges in cases where SSGD diverges for large learning rates in the large batch setting.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Generalization Guarantee of SGD for Pairwise Learning
We develop a novel high-probability generalization bound for uniformly-stable algorithms to incorporate the variance information for better generalization, based on which we establish the first nonsmooth learning algorithm to achieve almost optimal high-probability and dimension-independent generalization bounds in linear time.
Asynchronous Decentralized Distributed Training of Acoustic Models
Specifically, we study three variants of asynchronous decentralized parallel SGD (ADPSGD), namely, fixed and randomized communication patterns on a ring as well as a delay-by-one scheme.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
On the Initialization for Convex-Concave Min-max Problems
This means that the convergence speed does not have any improvement even if the algorithm starts from the optimal solution, and hence, is oblivious to the initialization.
On the Last Iterate Convergence of Momentum Methods
In this paper, we focus on the convergence rate of the last iterate of SGDM.
Why Does Decentralized Training Outperform Synchronous Training In The Large Batch Setting?
We conduct extensive studies over 12 state-of-the-art DL models/tasks and demonstrate that DPSGD consistently outperforms SSGD in the large batch setting; and DPSGD converges in cases where SSGD diverges for large learning rates.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Adam$^+$: A Stochastic Method with Adaptive Variance Reduction
As a result, Adam$^+$ requires few parameter tuning, as Adam, but it enjoys a provable convergence guarantee.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
Communication-Efficient Distributed Stochastic AUC Maximization with Deep Neural Networks
In this paper, we study distributed algorithms for large-scale AUC maximization with a deep neural network as a predictive model.
Improving Efficiency in Large-Scale Decentralized Distributed Training
Decentralized Parallel SGD (D-PSGD) and its asynchronous variant Asynchronous Parallel SGD (AD-PSGD) is a family of distributed learning algorithms that have been demonstrated to perform well for large-scale deep learning tasks.
Attacking Lifelong Learning Models with Gradient Reversion
We evaluate the effectiveness of traditional attack methods such as FGSM and PGD. The results show that A-GEM still possesses strong continual learning ability in the presence of adversarial examples in the memory and simple defense techniques such as label smoothing can further alleviate the adversarial effects.
Towards Better Understanding of Adaptive Gradient Algorithms in Generative Adversarial Nets
Then we propose an adaptive variant of OSG named Optimistic Adagrad (OAdagrad) and reveal an \emph{improved} adaptive complexity $O\left(\epsilon^{-\frac{2}{1-\alpha}}\right)$, where $\alpha$ characterizes the growth rate of the cumulative stochastic gradient and $0\leq \alpha\leq 1/2$.
A Decentralized Parallel Algorithm for Training Generative Adversarial Nets
Despite recent progress on decentralized algorithms for training deep neural networks, it remains unclear whether it is possible to train GANs in a decentralized manner.
Improved Schemes for Episodic Memory-based Lifelong Learning
This view leads to two improved schemes for episodic memory based lifelong learning, called MEGA-I and MEGA-II.
Learning with Long-term Remembering: Following the Lead of Mixed Stochastic Gradient
In this paper, we introduce a novel and effective lifelong learning algorithm, called MixEd stochastic GrAdient (MEGA), which allows deep neural networks to acquire the ability of retaining performance on old tasks while learning new tasks.
Stochastic AUC Maximization with Deep Neural Networks
In this paper, we consider stochastic AUC maximization problem with a deep neural network as the predictive model.
Adaptive Negative Curvature Descent with Applications in Non-convex Optimization
Negative curvature descent (NCD) method has been utilized to design deterministic or stochastic algorithms for non-convex optimization aiming at finding second-order stationary points or local minima.
Faster Online Learning of Optimal Threshold for Consistent F-measure Optimization
To advance OFO, we propose an efficient online algorithm based on simultaneously learning a posterior probability of class and learning an optimal threshold by minimizing a stochastic strongly convex function with unknown strong convexity parameter.
First-order Convergence Theory for Weakly-Convex-Weakly-Concave Min-max Problems
In this paper, we consider first-order convergence theory and algorithms for solving a class of non-convex non-concave min-max saddle-point problems, whose objective function is weakly convex in the variables of minimization and weakly concave in the variables of maximization.
Weakly-Convex Concave Min-Max Optimization: Provable Algorithms and Applications in Machine Learning
Min-max problems have broad applications in machine learning, including learning with non-decomposable loss and learning with robustness to data distribution.
Fast Stochastic AUC Maximization with $O(1/n)$-Convergence Rate
In this paper, we consider statistical learning with AUC (area under ROC curve) maximization in the classical stochastic setting where one random data drawn from an unknown distribution is revealed at each iteration for updating the model.
Fast Rates of ERM and Stochastic Approximation: Adaptive to Error Bound Conditions
Error bound conditions (EBC) are properties that characterize the growth of an objective function when a point is moved away from the optimal set.
Adaptive Accelerated Gradient Converging Method under H\"{o}lderian Error Bound Condition
Recent studies have shown that proximal gradient (PG) method and accelerated gradient method (APG) with restarting can enjoy a linear convergence under a weaker condition than strong convexity, namely a quadratic growth condition (QGC).
ADMM without a Fixed Penalty Parameter: Faster Convergence with New Adaptive Penalization
The novelty of the proposed scheme lies at that it is adaptive to a local sharpness property of the objective function, which marks the key difference from previous adaptive scheme that adjusts the penalty parameter per-iteration based on certain conditions on iterates.
Stochastic Non-convex Optimization with Strong High Probability Second-order Convergence
In this paper, we study stochastic non-convex optimization with non-convex random functions.
On Noisy Negative Curvature Descent: Competing with Gradient Descent for Faster Non-convex Optimization
To the best of our knowledge, the proposed stochastic algorithm is the first one that converges to a second-order stationary point in {\it high probability} with a time complexity independent of the sample size and almost linear in dimensionality.
Adaptive Accelerated Gradient Converging Methods under Holderian Error Bound Condition
Recent studies have shown that proximal gradient (PG) method and accelerated gradient method (APG) with restarting can enjoy a linear convergence under a weaker condition than strong convexity, namely a quadratic growth condition (QGC).
