Search Results for author:
Found 50 papers, 9 papers with code
Stacking as Accelerated Gradient Descent
Stacking, a heuristic technique for training deep residual networks by progressively increasing the number of layers and initializing new layers by copying parameters from older layers, has proven quite successful in improving the efficiency of training deep neural networks.
Efficient Stagewise Pretraining via Progressive Subnetworks
We propose an instantiation of this framework - Random Part Training (RAPTR) - that selects and trains only a random subnetwork (e. g. depth-wise, width-wise) of the network at each step, progressively increasing the size in stages.
Asynchronous Local-SGD Training for Language Modeling
Local stochastic gradient descent (Local-SGD), also referred to as federated averaging, is an approach to distributed optimization where each device performs more than one SGD update per communication.
Improved Differentially Private and Lazy Online Convex Optimization
We study the task of $(\epsilon, \delta)$-differentially private online convex optimization (OCO).
On the Convergence of Federated Averaging with Cyclic Client Participation
Federated Averaging (FedAvg) and its variants are the most popular optimization algorithms in federated learning (FL).
From Gradient Flow on Population Loss to Learning with Stochastic Gradient Descent
When these potentials further satisfy certain self-bounding properties, we show that they can be used to provide a convergence guarantee for Gradient Descent (GD) and SGD (even when the paths of GF and GD/SGD are quite far apart).
Private Matrix Approximation and Geometry of Unitary Orbits
Consider the following optimization problem: Given $n \times n$ matrices $A$ and $\Lambda$, maximize $\langle A, U\Lambda U^*\rangle$ where $U$ varies over the unitary group $\mathrm{U}(n)$.
Beyond Uniform Lipschitz Condition in Differentially Private Optimization
Most prior results on differentially private stochastic gradient descent (DP-SGD) are derived under the simplistic assumption of uniform Lipschitzness, i. e., the per-sample gradients are uniformly bounded.
On the Unreasonable Effectiveness of Federated Averaging with Heterogeneous Data
Motivated by this observation, we propose a new quantity, average drift at optimum, to measure the effects of data heterogeneity, and explicitly use it to present a new theoretical analysis of FedAvg.
Self-Consistency of the Fokker-Planck Equation
In this paper, we exploit this concept to design a potential function of the hypothesis velocity fields, and prove that, if such a function diminishes to zero during the training procedure, the trajectory of the densities generated by the hypothesis velocity fields converges to the solution of the FPE in the Wasserstein-2 sense.
Mixed Federated Learning: Joint Decentralized and Centralized Learning
For example, additional datacenter data can be leveraged to jointly learn from centralized (datacenter) and decentralized (federated) training data and better match an expected inference data distribution.
Reproducibility in Optimization: Theoretical Framework and Limits
We initiate a formal study of reproducibility in optimization.
Pushing the Efficiency-Regret Pareto Frontier for Online Learning of Portfolios and Quantum States
This algorithm, called SCHRODINGER'S BISONS, is the first efficient algorithm with polylogarithmic regret for this more general problem.
Agnostic Learnability of Halfspaces via Logistic Loss
In this paper, we close this gap by constructing a well-behaved distribution such that the global minimizer of the logistic risk over this distribution only achieves $\Omega(\sqrt{\textrm{OPT}})$ misclassification risk, matching the upper bound in (Frei et al., 2021).
Breaking the centralized barrier for cross-device federated learning
Federated learning (FL) is a challenging setting for optimization due to the heterogeneity of the data across different clients which gives rise to the client drift phenomenon.
Efficient Methods for Online Multiclass Logistic Regression
Previous work (Foster et al., 2018) has highlighted the importance of improper predictors for achieving "fast rates" in the online multiclass logistic regression problem without suffering exponentially from secondary problem parameters, such as the norm of the predictors in the comparison class.
A Field Guide to Federated Optimization
Federated learning and analytics are a distributed approach for collaboratively learning models (or statistics) from decentralized data, motivated by and designed for privacy protection.
SGD: The Role of Implicit Regularization, Batch-size and Multiple-epochs
We show that there is an SCO problem such that GD with any step size and number of iterations can only learn at a suboptimal rate: at least $\widetilde{\Omega}(1/n^{5/12})$.
Federated Functional Gradient Boosting
First, in the semi-heterogeneous setting, when the marginal distributions of the feature vectors on client machines are identical, we develop the federated functional gradient boosting (FFGB) method that provably converges to the global minimum.
A Multiclass Boosting Framework for Achieving Fast and Provable Adversarial Robustness
This apparent lack of robustness has led researchers to propose methods that can help to prevent an adversary from having such capabilities.
Learning with User-Level Privacy
We show that for high-dimensional mean estimation, empirical risk minimization with smooth losses, stochastic convex optimization, and learning hypothesis classes with finite metric entropy, the privacy cost decreases as $O(1/\sqrt{m})$ as users provide more samples.
PAC-Bayes Learning Bounds for Sample-Dependent Priors
We present a series of new PAC-Bayes learning guarantees for randomized algorithms with sample-dependent priors.
Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning
Federated learning (FL) is a challenging setting for optimization due to the heterogeneity of the data across different clients which gives rise to the client drift phenomenon.
Estimating Training Data Influence by Tracing Gradient Descent
We introduce a method called TracIn that computes the influence of a training example on a prediction made by the model.
A Deep Conditioning Treatment of Neural Networks
We study the role of depth in training randomly initialized overparameterized neural networks.
Breaking the Glass Ceiling for Embedding-Based Classifiers for Large Output Spaces
In extreme classification settings, embedding-based neural network models are currently not competitive with sparse linear and tree-based methods in terms of accuracy.
SCAFFOLD: Stochastic Controlled Averaging for Federated Learning
We obtain tight convergence rates for FedAvg and prove that it suffers from `client-drift' when the data is heterogeneous (non-iid), resulting in unstable and slow convergence.
On the Convergence of Adam and Beyond
Several recently proposed stochastic optimization methods that have been successfully used in training deep networks such as RMSProp, Adam, Adadelta, Nadam are based on using gradient updates scaled by square roots of exponential moving averages of squared past gradients.
Hypothesis Set Stability and Generalization
Our main result is a generalization bound for data-dependent hypothesis sets expressed in terms of a notion of hypothesis set stability and a notion of Rademacher complexity for data-dependent hypothesis sets that we introduce.
Escaping Saddle Points with Adaptive Gradient Methods
Adaptive methods such as Adam and RMSProp are widely used in deep learning but are not well understood.
Adaptive Methods for Nonconvex Optimization
In this work, we provide a new analysis of such methods applied to nonconvex stochastic optimization problems, characterizing the effect of increasing minibatch size.
Stochastic Negative Mining for Learning with Large Output Spaces
Furthermore, we identify a particularly intuitive class of loss functions in the aforementioned family and show that they are amenable to practical implementation in the large output space setting (i. e. computation is possible without evaluating scores of all labels) by developing a technique called Stochastic Negative Mining.
Loss Decomposition for Fast Learning in Large Output Spaces
For problems with large output spaces, evaluation of the loss function and its gradient are expensive, typically taking linear time in the size of the output space.
Logistic Regression: The Importance of Being Improper
Starting with the simple observation that the logistic loss is $1$-mixable, we design a new efficient improper learning algorithm for online logistic regression that circumvents the aforementioned lower bound with a regret bound exhibiting a doubly-exponential improvement in dependence on the predictor norm.
Online Learning of Quantum States
Even in the "non-realizable" setting---where there could be arbitrary noise in the measurement outcomes---we show how to output hypothesis states that do significantly worse than the best possible states at most $\operatorname{O}\!\left(\sqrt {Tn}\right) $ times on the first $T$ measurements.
Parameter-free online learning via model selection
We introduce an efficient algorithmic framework for model selection in online learning, also known as parameter-free online learning.
Adaptive Feature Selection: Computationally Efficient Online Sparse Linear Regression under RIP
Online sparse linear regression is an online problem where an algorithm repeatedly chooses a subset of coordinates to observe in an adversarially chosen feature vector, makes a real-valued prediction, receives the true label, and incurs the squared loss.
Online Sparse Linear Regression
We consider the online sparse linear regression problem, which is the problem of sequentially making predictions observing only a limited number of features in each round, to minimize regret with respect to the best sparse linear regressor, where prediction accuracy is measured by square loss.
Hardness of Online Sleeping Combinatorial Optimization Problems
We show that several online combinatorial optimization problems that admit efficient no-regret algorithms become computationally hard in the sleeping setting where a subset of actions becomes unavailable in each round.
Online Gradient Boosting
We extend the theory of boosting for regression problems to the online learning setting.
Optimal and Adaptive Algorithms for Online Boosting
We study online boosting, the task of converting any weak online learner into a strong online learner.
Taming the Monster: A Fast and Simple Algorithm for Contextual Bandits
We present a new algorithm for the contextual bandit learning problem, where the learner repeatedly takes one of $K$ actions in response to the observed context, and observes the reward only for that chosen action.
Adaptive Market Making via Online Learning
We consider the design of strategies for \emph{market making} in a market like a stock, commodity, or currency exchange.
Multiarmed Bandits With Limited Expert Advice
We solve the COLT 2013 open problem of \citet{SCB} on minimizing regret in the setting of advice-efficient multiarmed bandits with expert advice.
Bargaining for Revenue Shares on Tree Trading Networks
We study trade networks with a tree structure, where a seller with a single indivisible good is connected to buyers, each with some value for the good, via a unique path of intermediaries.
Newtron: an Efficient Bandit algorithm for Online Multiclass Prediction
We prove that the regret of \newtron is \(O(\log T)\) when \(\alpha\) is a constant that does not vary with horizon \(T\), and at most \(O(T^{2/3})\) if \(\alpha\) is allowed to increase to infinity with \(T\).
Non-Stochastic Bandit Slate Problems
We consider bandit problems, motivated by applications in online advertising and news story selection, in which the learner must repeatedly select a slate, that is, a subset of size s from K possible actions, and then receives rewards for just the selected actions.
Beyond Convexity: Online Submodular Minimization
We consider an online decision problem over a discrete space in which the loss function is submodular.
On Stochastic and Worst-case Models for Investing
In practice, most investing is done assuming a probabilistic model of stock price returns known as the Geometric Brownian Motion (GBM).
Computational Equivalence of Fixed Points and No Regret Algorithms, and Convergence to Equilibria
We study the relation between notions of game-theoretic equilibria which are based on stability under a set of deviations, and empirical equilibria which are reached by rational players.
