Search Results for author:
Found 63 papers, 8 papers with code
In-Context Learning with Transformers: Softmax Attention Adapts to Function Lipschitzness
We show that an attention unit learns a window that it uses to implement a nearest-neighbors predictor adapted to the landscape of the pretraining tasks.
An Accelerated Gradient Method for Simple Bilevel Optimization with Convex Lower-level Problem
In this paper, we focus on simple bilevel optimization problems, where we minimize a convex smooth objective function over the optimal solution set of another convex smooth constrained optimization problem.
Krylov Cubic Regularized Newton: A Subspace Second-Order Method with Dimension-Free Convergence Rate
In this paper, we introduce a novel subspace cubic regularized Newton method that achieves a dimension-independent global convergence rate of ${O}\left(\frac{1}{mk}+\frac{1}{k^2}\right)$ for solving convex optimization problems.
Provable Multi-Task Representation Learning by Two-Layer ReLU Neural Networks
An increasingly popular machine learning paradigm is to pretrain a neural network (NN) on many tasks offline, then adapt it to downstream tasks, often by re-training only the last linear layer of the network.
Limited-Memory Greedy Quasi-Newton Method with Non-asymptotic Superlinear Convergence Rate
Interestingly, our established non-asymptotic superlinear convergence rate demonstrates an explicit trade-off between the convergence speed and memory requirement, which to our knowledge, is the first of its kind.
Online Learning Guided Curvature Approximation: A Quasi-Newton Method with Global Non-Asymptotic Superlinear Convergence
Quasi-Newton algorithms are among the most popular iterative methods for solving unconstrained minimization problems, largely due to their favorable superlinear convergence property.
InfoNCE Loss Provably Learns Cluster-Preserving Representations
The goal of contrasting learning is to learn a representation that preserves underlying clusters by keeping samples with similar content, e. g. the ``dogness'' of a dog, close to each other in the space generated by the representation.
Network Adaptive Federated Learning: Congestion and Lossy Compression
In order to achieve the dual goals of privacy and learning across distributed data, Federated Learning (FL) systems rely on frequent exchanges of large files (model updates) between a set of clients and the server.
Future Gradient Descent for Adapting the Temporal Shifting Data Distribution in Online Recommendation Systems
One of the key challenges of learning an online recommendation model is the temporal domain shift, which causes the mismatch between the training and testing data distribution and hence domain generalization error.
A Conditional Gradient-based Method for Simple Bilevel Optimization with Convex Lower-level Problem
To the best of our knowledge, our method achieves the best-known iteration complexity for the considered class of bilevel problems.
Straggler-Resilient Personalized Federated Learning
Federated Learning is an emerging learning paradigm that allows training models from samples distributed across a large network of clients while respecting privacy and communication restrictions.
FedAvg with Fine Tuning: Local Updates Lead to Representation Learning
We show that the reason behind generalizability of the FedAvg's output is its power in learning the common data representation among the clients' tasks, by leveraging the diversity among client data distributions via local updates.
Generalized Optimistic Methods for Convex-Concave Saddle Point Problems
In this paper, we follow this approach and distill the underlying idea of optimism to propose a generalized optimistic method, which includes the optimistic gradient method as a special case.
The Power of Adaptivity in SGD: Self-Tuning Step Sizes with Unbounded Gradients and Affine Variance
We study convergence rates of AdaGrad-Norm as an exemplar of adaptive stochastic gradient methods (SGD), where the step sizes change based on observed stochastic gradients, for minimizing non-convex, smooth objectives.
MAML and ANIL Provably Learn Representations
Recent empirical evidence has driven conventional wisdom to believe that gradient-based meta-learning (GBML) methods perform well at few-shot learning because they learn an expressive data representation that is shared across tasks.
Minimax Optimization: The Case of Convex-Submodular
Prior literature has thus far mainly focused on studying such problems in the continuous domain, e. g., convex-concave minimax optimization is now understood to a significant extent.
Exploiting Local Convergence of Quasi-Newton Methods Globally: Adaptive Sample Size Approach
In this paper, we use an adaptive sample size scheme that exploits the superlinear convergence of quasi-Newton methods globally and throughout the entire learning process.
Exploiting Shared Representations for Personalized Federated Learning
Based on this intuition, we propose a novel federated learning framework and algorithm for learning a shared data representation across clients and unique local heads for each client.
Generalization of Model-Agnostic Meta-Learning Algorithms: Recurring and Unseen Tasks
In this paper, we study the generalization properties of Model-Agnostic Meta-Learning (MAML) algorithms for supervised learning problems.
Straggler-Resilient Federated Learning: Leveraging the Interplay Between Statistical Accuracy and System Heterogeneity
Federated Learning is a novel paradigm that involves learning from data samples distributed across a large network of clients while the data remains local.
Personalized Federated Learning with Theoretical Guarantees: A Model-Agnostic Meta-Learning Approach
In this paper, we study a personalized variant of the federated learning in which our goal is to find an initial shared model that current or new users can easily adapt to their local dataset by performing one or a few steps of gradient descent with respect to their own data.
Second Order Optimality in Decentralized Non-Convex Optimization via Perturbed Gradient Tracking
In this paper we study the problem of escaping from saddle points and achieving second-order optimality in a decentralized setting where a group of agents collaborate to minimize their aggregate objective function.
How Does the Task Landscape Affect MAML Performance?
Model-Agnostic Meta-Learning (MAML) has become increasingly popular for training models that can quickly adapt to new tasks via one or few stochastic gradient descent steps.
Submodular Meta-Learning
Motivated by this terminology, we propose a novel meta-learning framework in the discrete domain where each task is equivalent to maximizing a set function under a cardinality constraint.
Federated Learning with Compression: Unified Analysis and Sharp Guarantees
In federated learning, communication cost is often a critical bottleneck to scale up distributed optimization algorithms to collaboratively learn a model from millions of devices with potentially unreliable or limited communication and heterogeneous data distributions.
Safe Learning under Uncertain Objectives and Constraints
More precisely, by assuming that Reliable-FW has access to a (stochastic) gradient oracle of the objective function and a noisy feasibility oracle of the safety polytope, it finds an $\epsilon$-approximate first-order stationary point with the optimal ${\mathcal{O}}({1}/{\epsilon^2})$ gradient oracle complexity (resp.
Hybrid Model for Anomaly Detection on Call Detail Records by Time Series Forecasting
In this paper, a new hybrid method is proposed based on various anomaly detection methods such as GARCH, K-means, and Neural Network to determine the anomalous data.
Non-asymptotic Superlinear Convergence of Standard Quasi-Newton Methods
In this paper, we provide a finite-time (non-asymptotic) convergence analysis for Broyden quasi-Newton algorithms under the assumptions that the objective function is strongly convex, its gradient is Lipschitz continuous, and its Hessian is Lipschitz continuous at the optimal solution.
Quantized Decentralized Stochastic Learning over Directed Graphs
We consider a decentralized stochastic learning problem where data points are distributed among computing nodes communicating over a directed graph.
Personalized Federated Learning: A Meta-Learning Approach
In this paper, we study a personalized variant of the federated learning in which our goal is to find an initial shared model that current or new users can easily adapt to their local dataset by performing one or a few steps of gradient descent with respect to their own data.
On the Convergence Theory of Debiased Model-Agnostic Meta-Reinforcement Learning
We consider Model-Agnostic Meta-Learning (MAML) methods for Reinforcement Learning (RL) problems, where the goal is to find a policy using data from several tasks represented by Markov Decision Processes (MDPs) that can be updated by one step of stochastic policy gradient for the realized MDP.
Task-Robust Model-Agnostic Meta-Learning
Meta-learning methods have shown an impressive ability to train models that rapidly learn new tasks.
Stochastic Continuous Greedy ++: When Upper and Lower Bounds Match
Concretely, for a monotone and continuous DR-submodular function, \SCGPP achieves a tight $[(1-1/e)\OPT -\epsilon]$ solution while using $O(1/\epsilon^2)$ stochastic gradients and $O(1/\epsilon)$ calls to the linear optimization oracle.
A Decentralized Proximal Point-type Method for Saddle Point Problems
In this paper, we focus on solving a class of constrained non-convex non-concave saddle point problems in a decentralized manner by a group of nodes in a network.
One Sample Stochastic Frank-Wolfe
One of the beauties of the projected gradient descent method lies in its rather simple mechanism and yet stable behavior with inexact, stochastic gradients, which has led to its wide-spread use in many machine learning applications.
FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantization
Federated learning is a distributed framework according to which a model is trained over a set of devices, while keeping data localized.
On the Convergence Theory of Gradient-Based Model-Agnostic Meta-Learning Algorithms
We study the convergence of a class of gradient-based Model-Agnostic Meta-Learning (MAML) methods and characterize their overall complexity as well as their best achievable accuracy in terms of gradient norm for nonconvex loss functions.
Robust and Communication-Efficient Collaborative Learning
We consider a decentralized learning problem, where a set of computing nodes aim at solving a non-convex optimization problem collaboratively.
Convergence Rate of $\mathcal{O}(1/k)$ for Optimistic Gradient and Extra-gradient Methods in Smooth Convex-Concave Saddle Point Problems
To do so, we first show that both OGDA and EG can be interpreted as approximate variants of the proximal point method.
Stochastic Conditional Gradient++
It is known that this rate is optimal in terms of stochastic gradient evaluations.
Quantized Frank-Wolfe: Faster Optimization, Lower Communication, and Projection Free
How can we efficiently mitigate the overhead of gradient communications in distributed optimization?
A Unified Analysis of Extra-gradient and Optimistic Gradient Methods for Saddle Point Problems: Proximal Point Approach
In this paper we consider solving saddle point problems using two variants of Gradient Descent-Ascent algorithms, Extra-gradient (EG) and Optimistic Gradient Descent Ascent (OGDA) methods.
Efficient Distributed Hessian Free Algorithm for Large-scale Empirical Risk Minimization via Accumulating Sample Strategy
In this paper, we propose a Distributed Accumulated Newton Conjugate gradiEnt (DANCE) method in which sample size is gradually increasing to quickly obtain a solution whose empirical loss is under satisfactory statistical accuracy.
Escaping Saddle Points in Constrained Optimization
We propose a generic framework that yields convergence to a second-order stationary point of the problem, if the convex set $\mathcal{C}$ is simple for a quadratic objective function.
An Exact Quantized Decentralized Gradient Descent Algorithm
We consider the problem of decentralized consensus optimization, where the sum of $n$ smooth and strongly convex functions are minimized over $n$ distributed agents that form a connected network.
Towards More Efficient Stochastic Decentralized Learning: Faster Convergence and Sparse Communication
Recently, the decentralized optimization problem is attracting growing attention.
Direct Runge-Kutta Discretization Achieves Acceleration
We study gradient-based optimization methods obtained by directly discretizing a second-order ordinary differential equation (ODE) related to the continuous limit of Nesterov's accelerated gradient method.
Stochastic Conditional Gradient Methods: From Convex Minimization to Submodular Maximization
Further, for a monotone and continuous DR-submodular function and subject to a general convex body constraint, we prove that our proposed method achieves a $((1-1/e)OPT-\eps)$ guarantee with $O(1/\eps^3)$ stochastic gradient computations.
Conditional Gradient Method for Stochastic Submodular Maximization: Closing the Gap
More precisely, for a monotone and continuous DR-submodular function and subject to a \textit{general} convex body constraint, we prove that \alg achieves a $[(1-1/e)\text{OPT} -\eps]$ guarantee (in expectation) with $\mathcal{O}{(1/\eps^3)}$ stochastic gradient computations.
First-Order Adaptive Sample Size Methods to Reduce Complexity of Empirical Risk Minimization
Theoretical analyses show that the use of adaptive sample size methods reduces the overall computational cost of achieving the statistical accuracy of the whole dataset for a broad range of deterministic and stochastic first-order methods.
Large Scale Empirical Risk Minimization via Truncated Adaptive Newton Method
In this paper, we propose a novel adaptive sample size second-order method, which reduces the cost of computing the Hessian by solving a sequence of ERM problems corresponding to a subset of samples and lowers the cost of computing the Hessian inverse using a truncated eigenvalue decomposition.
IQN: An Incremental Quasi-Newton Method with Local Superlinear Convergence Rate
This makes their computational cost per iteration independent of the number of objective functions $n$.
Surpassing Gradient Descent Provably: A Cyclic Incremental Method with Linear Convergence Rate
We prove that not only the proposed DIAG method converges linearly to the optimal solution, but also its linear convergence factor justifies the advantage of incremental methods on GD.
Stochastic Averaging for Constrained Optimization with Application to Online Resource Allocation
Existing approaches to resource allocation for nowadays stochastic networks are challenged to meet fast convergence and tolerable delay requirements.
A Class of Parallel Doubly Stochastic Algorithms for Large-Scale Learning
Algorithms that are parallel in either of these dimensions exist, but RAPSA is the first attempt at a methodology that is parallel in both the selection of blocks and the selection of elements of the training set.
Adaptive Newton Method for Empirical Risk Minimization to Statistical Accuracy
We consider empirical risk minimization for large-scale datasets.
A Decentralized Quasi-Newton Method for Dual Formulations of Consensus Optimization
The resulting dual D-BFGS method is a fully decentralized algorithm in which nodes approximate curvature information of themselves and their neighbors through the satisfaction of a secant condition.
Doubly Random Parallel Stochastic Methods for Large Scale Learning
Algorithms that are parallel in either of these dimensions exist, but RAPSA is the first attempt at a methodology that is parallel in both, the selection of blocks and the selection of elements of the training set.
Online Optimization in Dynamic Environments: Improved Regret Rates for Strongly Convex Problems
In this paper, we address tracking of a time-varying parameter with unknown dynamics.
DSA: Decentralized Double Stochastic Averaging Gradient Algorithm
The decentralized double stochastic averaging gradient (DSA) algorithm is proposed as a solution alternative that relies on: (i) The use of local stochastic averaging gradients.
Optimization and Control
Global Convergence of Online Limited Memory BFGS
Global convergence of an online (stochastic) limited memory version of the Broyden-Fletcher- Goldfarb-Shanno (BFGS) quasi-Newton method for solving optimization problems with stochastic objectives that arise in large scale machine learning is established.
A Quasi-Newton Method for Large Scale Support Vector Machines
This paper adapts a recently developed regularized stochastic version of the Broyden, Fletcher, Goldfarb, and Shanno (BFGS) quasi-Newton method for the solution of support vector machine classification problems.
RES: Regularized Stochastic BFGS Algorithm
Numerical experiments showcase reductions in convergence time relative to stochastic gradient descent algorithms and non-regularized stochastic versions of BFGS.
