Search Results for author:
Found 38 papers, 8 papers with code
Stacking as Accelerated Gradient Descent
Stacking, a heuristic technique for training deep residual networks by progressively increasing the number of layers and initializing new layers by copying parameters from older layers, has proven quite successful in improving the efficiency of training deep neural networks.
Towards Quantifying the Preconditioning Effect of Adam
Specifically, for a $d$-dimensional quadratic with a diagonal Hessian having condition number $\kappa$, we show that the effective condition number-like quantity controlling the iteration complexity of Adam without momentum is $\mathcal{O}(\min(d, \kappa))$.
Improved Differentially Private and Lazy Online Convex Optimization
We study the task of $(\epsilon, \delta)$-differentially private online convex optimization (OCO).
Spectral State Space Models
This paper studies sequence modeling for prediction tasks with long range dependencies.
HAVE-Net: Hallucinated Audio-Visual Embeddings for Few-Shot Classification with Unimodal Cues
Even though joint training of audio-visual modalities improves classification performance in a low-data regime, it has yet to be thoroughly investigated in the RS domain.
Benchmarking Neural Network Training Algorithms
In order to address these challenges, we introduce a new, competitive, time-to-result benchmark using multiple workloads running on fixed hardware, the AlgoPerf: Training Algorithms benchmark.
Variance-Reduced Conservative Policy Iteration
We study the sample complexity of reducing reinforcement learning to a sequence of empirical risk minimization problems over the policy space.
Best of Both Worlds in Online Control: Competitive Ratio and Policy Regret
We consider the fundamental problem of online control of a linear dynamical system from two different viewpoints: regret minimization and competitive analysis.
Multi-User Reinforcement Learning with Low Rank Rewards
In this work, we consider the problem of collaborative multi-user reinforcement learning.
Adaptive Gradient Methods at the Edge of Stability
Very little is known about the training dynamics of adaptive gradient methods like Adam in deep learning.
Pushing the Efficiency-Regret Pareto Frontier for Online Learning of Portfolios and Quantum States
This algorithm, called SCHRODINGER'S BISONS, is the first efficient algorithm with polylogarithmic regret for this more general problem.
Machine Learning for Mechanical Ventilation Control (Extended Abstract)
Mechanical ventilation is one of the most widely used therapies in the ICU.
Online Target Q-learning with Reverse Experience Replay: Efficiently finding the Optimal Policy for Linear MDPs
The starting point of our work is the observation that in practice, Q-learning is used with two important modifications: (i) training with two networks, called online network and target network simultaneously (online target learning, or OTL) , and (ii) experience replay (ER) (Mnih et al., 2015).
The Skellam Mechanism for Differentially Private Federated Learning
We introduce the multi-dimensional Skellam mechanism, a discrete differential privacy mechanism based on the difference of two independent Poisson random variables.
Efficient Methods for Online Multiclass Logistic Regression
Previous work (Foster et al., 2018) has highlighted the importance of improper predictors for achieving "fast rates" in the online multiclass logistic regression problem without suffering exponentially from secondary problem parameters, such as the norm of the predictors in the comparison class.
Learning Rate Grafting: Transferability of Optimizer Tuning
In the empirical science of training large neural networks, the learning rate schedule is a notoriously challenging-to-tune hyperparameter, which can depend on all other properties (architecture, optimizer, batch size, dataset, regularization, ...) of the problem.
Acceleration via Fractal Learning Rate Schedules
In practical applications of iterative first-order optimization, the learning rate schedule remains notoriously difficult to understand and expensive to tune.
A Regret Minimization Approach to Iterative Learning Control
We consider the setting of iterative learning control, or model-based policy learning in the presence of uncertain, time-varying dynamics.
Deluca -- A Differentiable Control Library: Environments, Methods, and Benchmarking
We present an open-source library of natively differentiable physics and robotics environments, accompanied by gradient-based control methods and a benchmark-ing suite.
Machine Learning for Mechanical Ventilation Control
We consider the problem of controlling an invasive mechanical ventilator for pressure-controlled ventilation: a controller must let air in and out of a sedated patient's lungs according to a trajectory of airway pressures specified by a clinician.
Stochastic Optimization with Laggard Data Pipelines
State-of-the-art optimization is steadily shifting towards massively parallel pipelines with extremely large batch sizes.
Disentangling Adaptive Gradient Methods from Learning Rates
We investigate several confounding factors in the evaluation of optimization algorithms for deep learning.
A Deep Conditioning Treatment of Neural Networks
We study the role of depth in training randomly initialized overparameterized neural networks.
Revisiting the Generalization of Adaptive Gradient Methods
A commonplace belief in the machine learning community is that using adaptive gradient methods hurts generalization.
Logarithmic Regret for Online Control
We study optimal regret bounds for control in linear dynamical systems under adversarially changing strongly convex cost functions, given the knowledge of transition dynamics.
Boosting for Control of Dynamical Systems
We study the question of how to aggregate controllers for dynamical systems in order to improve their performance.
Online Control with Adversarial Disturbances
We study the control of a linear dynamical system with adversarial disturbances (as opposed to statistical noise).
Extreme Tensoring for Low-Memory Preconditioning
State-of-the-art models are now trained with billions of parameters, reaching hardware limits in terms of memory consumption.
Learning in Non-convex Games with an Optimization Oracle
We consider online learning in an adversarial, non-convex setting under the assumption that the learner has an access to an offline optimization oracle.
Efficient Full-Matrix Adaptive Regularization
Due to the large number of parameters of machine learning problems, full-matrix preconditioning methods are prohibitively expensive.
cpSGD: Communication-efficient and differentially-private distributed SGD
Distributed stochastic gradient descent is an important subroutine in distributed learning.
Optimal Sketching Bounds for Exp-concave Stochastic Minimization
We derive optimal statistical and computational complexity bounds for exp-concave stochastic minimization in terms of the effective dimension.
Leverage Score Sampling for Faster Accelerated Regression and ERM
Given a matrix $\mathbf{A}\in\mathbb{R}^{n\times d}$ and a vector $b \in\mathbb{R}^{d}$, we show how to compute an $\epsilon$-approximate solution to the regression problem $ \min_{x\in\mathbb{R}^{d}}\frac{1}{2} \|\mathbf{A} x - b\|_{2}^{2} $ in time $ \tilde{O} ((n+\sqrt{d\cdot\kappa_{\text{sum}}})\cdot s\cdot\log\epsilon^{-1}) $ where $\kappa_{\text{sum}}=\mathrm{tr}\left(\mathbf{A}^{\top}\mathbf{A}\right)/\lambda_{\min}(\mathbf{A}^{T}\mathbf{A})$ and $s$ is the maximum number of non-zero entries in a row of $\mathbf{A}$.
Lower Bounds for Higher-Order Convex Optimization
State-of-the-art methods in convex and non-convex optimization employ higher-order derivative information, either implicitly or explicitly.
The Price of Differential Privacy For Online Learning
We design differentially private algorithms for the problem of online linear optimization in the full information and bandit settings with optimal $\tilde{O}(\sqrt{T})$ regret bounds.
Finding Approximate Local Minima Faster than Gradient Descent
We design a non-convex second-order optimization algorithm that is guaranteed to return an approximate local minimum in time which scales linearly in the underlying dimension and the number of training examples.
Second-Order Stochastic Optimization for Machine Learning in Linear Time
First-order stochastic methods are the state-of-the-art in large-scale machine learning optimization owing to efficient per-iteration complexity.
Multisection in the Stochastic Block Model using Semidefinite Programming
We consider the problem of identifying underlying community-like structures in graphs.
