Search Results for author:
Found 27 papers, 16 papers with code
Optimization with access to auxiliary information
We investigate the fundamental optimization question of minimizing a target function $f(x)$ whose gradients are expensive to compute or have limited availability, given access to some auxiliary side function $h(x)$ whose gradients are cheap or more available.
Agree to Disagree: Diversity through Disagreement for Better Transferability
This behavior can hinder the transferability of trained models by (i) favoring the learning of simpler but spurious features -- present in the training data but absent from the test data -- and (ii) by only leveraging a small subset of predictive features.
Byzantine-Robust Decentralized Learning via Self-Centered Clipping
In this paper, we study the challenging task of Byzantine-robust decentralized training on arbitrary communication graphs.
Breaking the centralized barrier for cross-device federated learning
Federated learning (FL) is a challenging setting for optimization due to the heterogeneity of the data across different clients which gives rise to the client drift phenomenon.
Linear Speedup in Personalized Collaborative Learning
Collaborative training can improve the accuracy of a model for a user by trading off the model's bias (introduced by using data from other users who are potentially different) against its variance (due to the limited amount of data on any single user).
Towards Model Agnostic Federated Learning Using Knowledge Distillation
Is it possible to design an universal API for federated learning using which an ad-hoc group of data-holders (agents) collaborate with each other and perform federated learning?
RelaySum for Decentralized Deep Learning on Heterogeneous Data
A key challenge, primarily in decentralized deep learning, remains the handling of differences between the workers' local data distributions.
A Field Guide to Federated Optimization
Federated learning and analytics are a distributed approach for collaboratively learning models (or statistics) from decentralized data, motivated by and designed for privacy protection.
Quasi-Global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Data
In this paper, we investigate and identify the limitation of several decentralized optimization algorithms for different degrees of data heterogeneity.
Learning from History for Byzantine Robust Optimization
Secondly, we prove that even if the aggregation rules may succeed in limiting the influence of the attackers in a single round, the attackers can couple their attacks across time eventually leading to divergence.
Practical Low-Rank Communication Compression in Decentralized Deep Learning
Lossy gradient compression has become a practical tool to overcome the communication bottleneck in centrally coordinated distributed training of machine learning models.
Byzantine-Robust Learning on Heterogeneous Datasets via Resampling
In Byzantine-robust distributed optimization, a central server wants to train a machine learning model over data distributed across multiple workers.
Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning
Federated learning (FL) is a challenging setting for optimization due to the heterogeneity of the data across different clients which gives rise to the client drift phenomenon.
PowerGossip: Practical Low-Rank Communication Compression in Decentralized Deep Learning
Lossy gradient compression has become a practical tool to overcome the communication bottleneck in centrally coordinated distributed training of machine learning models.
Byzantine-Robust Learning on Heterogeneous Datasets via Bucketing
In Byzantine robust distributed or federated learning, a central server wants to train a machine learning model over data distributed across multiple workers.
Secure Byzantine-Robust Machine Learning
Increasingly machine learning systems are being deployed to edge servers and devices (e. g. mobile phones) and trained in a collaborative manner.
Why are Adaptive Methods Good for Attention Models?
While stochastic gradient descent (SGD) is still the \emph{de facto} algorithm in deep learning, adaptive methods like Clipped SGD/Adam have been observed to outperform SGD across important tasks, such as attention models.
SCAFFOLD: Stochastic Controlled Averaging for Federated Learning
We obtain tight convergence rates for FedAvg and prove that it suffers from `client-drift' when the data is heterogeneous (non-iid), resulting in unstable and slow convergence.
Why ADAM Beats SGD for Attention Models
While stochastic gradient descent (SGD) is still the de facto algorithm in deep learning, adaptive methods like Adam have been observed to outperform SGD across important tasks, such as attention models.
The Error-Feedback Framework: Better Rates for SGD with Delayed Gradients and Compressed Communication
We analyze (stochastic) gradient descent (SGD) with delayed updates on smooth quasi-convex and non-convex functions and derive concise, non-asymptotic, convergence rates.
Amplifying Rényi Differential Privacy via Shuffling
Differential privacy is a useful tool to build machine learning models which do not release too much information about the training data.
PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization
We study gradient compression methods to alleviate the communication bottleneck in data-parallel distributed optimization.
Accelerating Gradient Boosting Machine
This is the first GBM type of algorithm with theoretically-justified accelerated convergence rate.
Error Feedback Fixes SignSGD and other Gradient Compression Schemes
These issues arise because of the biased nature of the sign compression operator.
Efficient Greedy Coordinate Descent for Composite Problems
For these problems we provide (i) the first linear rates of convergence independent of $n$, and show that our greedy update rule provides speedups similar to those obtained in the smooth case.
Global linear convergence of Newton's method without strong-convexity or Lipschitz gradients
We show that Newton's method converges globally at a linear rate for objective functions whose Hessians are stable.
On Matching Pursuit and Coordinate Descent
Exploiting the connection between the two algorithms, we present a unified analysis of both, providing affine invariant sublinear $\mathcal{O}(1/t)$ rates on smooth objectives and linear convergence on strongly convex objectives.
