Search Results for author:
Found 49 papers, 12 papers with code
Parameter-free, Dynamic, and Strongly-Adaptive Online Learning
We provide a new online learning algorithm that for the first time combines several disparate notions of adaptivity.
Empirical Tests of Optimization Assumptions in Deep Learning
There is a significant gap between our theoretical understanding of optimization algorithms used in deep learning and their practical performance.
Private Zeroth-Order Nonsmooth Nonconvex Optimization
We introduce a new zeroth-order algorithm for private stochastic optimization on nonconvex and nonsmooth objectives.
Fully Unconstrained Online Learning
We provide an online learning algorithm that obtains regret $G\|w_\star\|\sqrt{T\log(\|w_\star\|G\sqrt{T})} + \|w_\star\|^2 + G^2$ on $G$-Lipschitz convex losses for any comparison point $w_\star$ without knowing either $G$ or $\|w_\star\|$.
Online Linear Regression in Dynamic Environments via Discounting
We develop algorithms for online linear regression which achieve optimal static and dynamic regret guarantees \emph{even in the complete absence of prior knowledge}.
Adam with model exponential moving average is effective for nonconvex optimization
In this work, we offer a theoretical analysis of two modern optimization techniques for training large and complex models: (i) adaptive optimization algorithms, such as Adam, and (ii) the model exponential moving average (EMA).
The Road Less Scheduled
Existing learning rate schedules that do not require specification of the optimization stopping step T are greatly out-performed by learning rate schedules that depend on T. We propose an approach that avoids the need for this stopping time by eschewing the use of schedules entirely, while exhibiting state-of-the-art performance compared to schedules across a wide family of problems ranging from convex problems to large-scale deep learning problems.
Random Scaling and Momentum for Non-smooth Non-convex Optimization
Training neural networks requires optimizing a loss function that may be highly irregular, and in particular neither convex nor smooth.
When, Why and How Much? Adaptive Learning Rate Scheduling by Refinement
To go beyond this worst-case analysis, we use the observed gradient norms to derive schedules refined for any particular task.
Improving Adaptive Online Learning Using Refined Discretization
Motivated by the pursuit of instance optimality, we propose a new algorithm that simultaneously achieves ($i$) the AdaGrad-style second order gradient adaptivity; and ($ii$) the comparator norm adaptivity also known as "parameter freeness" in the literature.
Unconstrained Online Learning with Unbounded Losses
Algorithms for online learning typically require one or more boundedness assumptions: that the domain is bounded, that the losses are Lipschitz, or both.
Optimal Stochastic Non-smooth Non-convex Optimization through Online-to-Non-convex Conversion
Our primary technique is a reduction from non-smooth non-convex optimization to online learning, after which our results follow from standard regret bounds in online learning.
Differentially Private Image Classification from Features
We find that linear regression is much more effective than logistic regression from both privacy and computational aspects, especially at stricter epsilon values ($\epsilon < 1$).
Ranked #34 on
Image Classification
on ImageNet
Parameter-free Regret in High Probability with Heavy Tails
We present new algorithms for online convex optimization over unbounded domains that obtain parameter-free regret in high-probability given access only to potentially heavy-tailed subgradient estimates.
Differentially Private Online-to-Batch for Smooth Losses
We develop a new reduction that converts any online convex optimization algorithm suffering $O(\sqrt{T})$ regret into an $\epsilon$-differentially private stochastic convex optimization algorithm with the optimal convergence rate $\tilde O(1/\sqrt{T} + \sqrt{d}/\epsilon T)$ on smooth losses in linear time, forming a direct analogy to the classical non-private "online-to-batch" conversion.
Momentum Aggregation for Private Non-convex ERM
We introduce new algorithms and convergence guarantees for privacy-preserving non-convex Empirical Risk Minimization (ERM) on smooth $d$-dimensional objectives.
Long Range Language Modeling via Gated State Spaces
State space models have shown to be effective at modeling long range dependencies, specially on sequence classification tasks.
Optimal Comparator Adaptive Online Learning with Switching Cost
Practical online learning tasks are often naturally defined on unconstrained domains, where optimal algorithms for general convex losses are characterized by the notion of comparator adaptivity.
Large Scale Transfer Learning for Differentially Private Image Classification
Moreover, by systematically comparing private and non-private models across a range of large batch sizes, we find that similar to non-private setting, choice of optimizer can further improve performance substantially with DP.
Implicit Parameter-free Online Learning with Truncated Linear Models
Parameter-free algorithms are online learning algorithms that do not require setting learning rates.
Leveraging Initial Hints for Free in Stochastic Linear Bandits
We study the setting of optimizing with bandit feedback with additional prior knowledge provided to the learner in the form of an initial hint of the optimal action.
Parameter-free Mirror Descent
We develop a modified online mirror descent framework that is suitable for building adaptive and parameter-free algorithms in unbounded domains.
Understanding AdamW through Proximal Methods and Scale-Freeness
First, we show how to re-interpret AdamW as an approximation of a proximal gradient method, which takes advantage of the closed-form proximal mapping of the regularizer instead of only utilizing its gradient information as in Adam-$\ell_2$.
PDE-Based Optimal Strategy for Unconstrained Online Learning
Unconstrained Online Linear Optimization (OLO) is a practical problem setting to study the training of machine learning models.
Logarithmic Regret from Sublinear Hints
We consider the online linear optimization problem, where at every step the algorithm plays a point $x_t$ in the unit ball, and suffers loss $\langle c_t, x_t\rangle$ for some cost vector $c_t$ that is then revealed to the algorithm.
Online Selective Classification with Limited Feedback
For example, this may model an adaptive decision to invoke more resources on this instance.
High-probability Bounds for Non-Convex Stochastic Optimization with Heavy Tails
We consider non-convex stochastic optimization using first-order algorithms for which the gradient estimates may have heavy tails.
Better SGD using Second-order Momentum
We develop a new algorithm for non-convex stochastic optimization that finds an $\epsilon$-critical point in the optimal $O(\epsilon^{-3})$ stochastic gradient and Hessian-vector product computations.
Adversarial Tracking Control via Strongly Adaptive Online Learning with Memory
Next, considering a related problem called online learning with memory, we construct a novel strongly adaptive algorithm that uses our first contribution as a building block.
Upper Confidence Bounds for Combining Stochastic Bandits
We provide a simple method to combine stochastic bandit algorithms.
Better Full-Matrix Regret via Parameter-Free Online Learning
We provide online convex optimization algorithms that guarantee improved full-matrix regret bounds.
Online Linear Optimization with Many Hints
We study an online linear optimization (OLO) problem in which the learner is provided access to $K$ "hint" vectors in each round prior to making a decision.
Extreme Memorization via Scale of Initialization
In the case of the homogeneous ReLU activation, we show that this behavior can be attributed to the loss function.
Comparator-adaptive Convex Bandits
We study bandit convex optimization methods that adapt to the norm of the comparator, a topic that has only been studied before for its full-information counterpart.
Online Learning with Imperfect Hints
We consider a variant of the classical online linear optimization problem in which at every step, the online player receives a "hint" vector before choosing the action for that round.
Adaptive Online Learning with Varying Norms
Given any increasing sequence of norms $\|\cdot\|_0,\dots,\|\cdot\|_{T-1}$, we provide an online convex optimization algorithm that outputs points $w_t$ in some domain $W$ in response to convex losses $\ell_t:W\to \mathbb{R}$ that guarantees regret $R_T(u)=\sum_{t=1}^T \ell_t(w_t)-\ell_t(u)\le \tilde O\left(\|u\|_{T-1}\sqrt{\sum_{t=1}^T \|g_t\|_{t-1,\star}^2}\right)$ where $g_t$ is a subgradient of $\ell_t$ at $w_t$.
Momentum Improves Normalized SGD
We provide an improved analysis of normalized SGD showing that adding momentum provably removes the need for large batch sizes on non-convex objectives.
Matrix-Free Preconditioning in Online Learning
We provide an online convex optimization algorithm with regret that interpolates between the regret of an algorithm using an optimal preconditioning matrix and one using a diagonal preconditioning matrix.
Kernel Truncated Randomized Ridge Regression: Optimal Rates and Low Noise Acceleration
In this paper, we consider the nonparametric least square regression in a Reproducing Kernel Hilbert Space (RKHS).
Momentum-Based Variance Reduction in Non-Convex SGD
Variance reduction has emerged in recent years as a strong competitor to stochastic gradient descent in non-convex problems, providing the first algorithms to improve upon the converge rate of stochastic gradient descent for finding first-order critical points.
Anytime Online-to-Batch Conversions, Optimism, and Acceleration
A standard way to obtain convergence guarantees in stochastic convex optimization is to run an online learning algorithm and then output the average of its iterates: the actual iterates of the online learning algorithm do not come with individual guarantees.
Artificial Constraints and Lipschitz Hints for Unconstrained Online Learning
We provide algorithms that guarantee regret $R_T(u)\le \tilde O(G\|u\|^3 + G(\|u\|+1)\sqrt{T})$ or $R_T(u)\le \tilde O(G\|u\|^3T^{1/3} + GT^{1/3}+ G\|u\|\sqrt{T})$ for online convex optimization with $G$-Lipschitz losses for any comparison point $u$ without prior knowledge of either $G$ or $\|u\|$.
Combining Online Learning Guarantees
We show how to take any two parameter-free online learning algorithms with different regret guarantees and obtain a single algorithm whose regret is the minimum of the two base algorithms.
Surrogate Losses for Online Learning of Stepsizes in Stochastic Non-Convex Optimization
Stochastic Gradient Descent (SGD) has played a central role in machine learning.
Black-Box Reductions for Parameter-free Online Learning in Banach Spaces
We introduce several new black-box reductions that significantly improve the design of adaptive and parameter-free online learning algorithms by simplifying analysis, improving regret guarantees, and sometimes even improving runtime.
Distributed Stochastic Optimization via Adaptive SGD
Scaling up the training process of these models is crucial, but the most popular algorithm, Stochastic Gradient Descent (SGD), is a serial method that is surprisingly hard to parallelize.
Stochastic and Adversarial Online Learning without Hyperparameters
Most online optimization algorithms focus on one of two things: performing well in adversarial settings by adapting to unknown data parameters (such as Lipschitz constants), typically achieving $O(\sqrt{T})$ regret, or performing well in stochastic settings where they can leverage some structure in the losses (such as strong convexity), typically achieving $O(\log(T))$ regret.
Online Convex Optimization with Unconstrained Domains and Losses
We propose an online convex optimization algorithm (RescaledExp) that achieves optimal regret in the unconstrained setting without prior knowledge of any bounds on the loss functions.
Online Learning Without Prior Information
The vast majority of optimization and online learning algorithms today require some prior information about the data (often in the form of bounds on gradients or on the optimal parameter value).
