Search Results for author:
Found 34 papers, 10 papers with code
Fishing For Cheap And Efficient Pruners At Initialization
Although the FIM-Hessian equivalency only holds for convergent models that maximize the likelihood, recent studies (Karakida et al., 2019) suggest that, even at initialization, the FIM captures essential geometric information of parameters in overparameterized NNs, providing the basis for our method.
Revisiting LocalSGD and SCAFFOLD: Improved Rates and Missing Analysis
LocalSGD and SCAFFOLD are widely used methods in distributed stochastic optimization, with numerous applications in machine learning, large-scale data processing, and federated learning.
Generalizing in Net-Zero Microgrids: A Study with Federated PPO and TRPO
This work addresses the challenge of optimal energy management in microgrids through a collaborative and privacy-preserving framework.
Methods with Local Steps and Random Reshuffling for Generally Smooth Non-Convex Federated Optimization
Many existing algorithms designed for standard smooth problems need to be revised.
FRUGAL: Memory-Efficient Optimization by Reducing State Overhead for Scalable Training
However, in all these algorithms, the $\textit{effective rank of the weight updates remains low-rank}$, which can lead to a substantial loss of information from the gradient.
Collaborative and Efficient Personalization with Mixtures of Adaptors
Non-iid data is prevalent in real-world federated learning problems.
Methods for Convex $(L_0,L_1)$-Smooth Optimization: Clipping, Acceleration, and Adaptivity
We also extend these results to the stochastic case under the over-parameterization assumption, propose a new accelerated method for convex $(L_0, L_1)$-smooth optimization, and derive new convergence rates for Adaptive Gradient Descent (Malitsky and Mishchenko, 2020).
Low-Resource Machine Translation through the Lens of Personalized Federated Learning
We present a new approach called MeritOpt based on the Personalized Federated Learning algorithm MeritFed that can be applied to Natural Language Tasks with heterogeneous data.
Clipping Improves Adam-Norm and AdaGrad-Norm when the Noise Is Heavy-Tailed
We also show that gradient clipping fixes this issue, i. e., we derive new high-probability convergence bounds with polylogarithmic dependence on the confidence level for AdaGrad-Norm and Adam-Norm with clipping and with/without delay for smooth convex/non-convex stochastic optimization with heavy-tailed noise.
Enhancing Policy Gradient with the Polyak Step-Size Adaption
Policy gradient is a widely utilized and foundational algorithm in the field of reinforcement learning (RL).
Generalized Policy Learning for Smart Grids: FL TRPO Approach
The smart grid domain requires bolstering the capabilities of existing energy management systems; Federated Learning (FL) aligns with this goal as it demonstrates a remarkable ability to train models on heterogeneous datasets while maintaining data privacy, making it suitable for smart grid applications, which often involve disparate data distributions and interdependencies among features that hinder the suitability of linear models.
Flashback: Understanding and Mitigating Forgetting in Federated Learning
In Federated Learning (FL), forgetting, or the loss of knowledge across rounds, hampers algorithm convergence, particularly in the presence of severe data heterogeneity among clients.
Federated Learning Can Find Friends That Are Advantageous
In Federated Learning (FL), the distributed nature and heterogeneity of client data present both opportunities and challenges.
Dirichlet-based Uncertainty Quantification for Personalized Federated Learning with Improved Posterior Networks
This paper presents a new approach to federated learning that allows selecting a model from global and personalized ones that would perform better for a particular input point.
Byzantine Robustness and Partial Participation Can Be Achieved at Once: Just Clip Gradient Differences
Distributed learning has emerged as a leading paradigm for training large machine learning models.
High-Probability Convergence for Composite and Distributed Stochastic Minimization and Variational Inequalities with Heavy-Tailed Noise
High-probability analysis of stochastic first-order optimization methods under mild assumptions on the noise has been gaining a lot of attention in recent years.
Clip21: Error Feedback for Gradient Clipping
Motivated by the increasing popularity and importance of large-scale training under differential privacy (DP) constraints, we study distributed gradient methods with gradient clipping, i. e., clipping applied to the gradients computed from local information at the nodes.
Global-QSGD: Practical Floatless Quantization for Distributed Learning with Theoretical Guarantees
To obtain theoretical guarantees, we generalize the notion of standard unbiased compression operators to incorporate Global-QSGD.
Partially Personalized Federated Learning: Breaking the Curse of Data Heterogeneity
We present a partially personalized formulation of Federated Learning (FL) that strikes a balance between the flexibility of personalization and cooperativeness of global training.
Federated Learning with Regularized Client Participation
Under this scheme, each client joins the learning process every $R$ communication rounds, which we refer to as a meta epoch.
High-Probability Bounds for Stochastic Optimization and Variational Inequalities: the Case of Unbounded Variance
During recent years the interest of optimization and machine learning communities in high-probability convergence of stochastic optimization methods has been growing.
PaDPaF: Partial Disentanglement with Partially-Federated GANs
Federated learning has become a popular machine learning paradigm with many potential real-life applications, including recommendation systems, the Internet of Things (IoT), healthcare, and self-driving cars.
Adaptive Learning Rates for Faster Stochastic Gradient Methods
In this work, we propose new adaptive step size strategies that improve several stochastic gradient methods.
Better Methods and Theory for Federated Learning: Compression, Client Selection and Heterogeneity
Federated learning (FL) is an emerging machine learning paradigm involving multiple clients, e. g., mobile phone devices, with an incentive to collaborate in solving a machine learning problem coordinated by a central server.
Variance Reduction is an Antidote to Byzantines: Better Rates, Weaker Assumptions and Communication Compression as a Cherry on the Top
However, many fruitful directions, such as the usage of variance reduction for achieving robustness and communication compression for reducing communication costs, remain weakly explored in the field.
FedShuffle: Recipes for Better Use of Local Work in Federated Learning
The practice of applying several local updates before aggregation across clients has been empirically shown to be a successful approach to overcoming the communication bottleneck in Federated Learning (FL).
FL_PyTorch: optimization research simulator for federated learning
Our system supports abstractions that provide researchers with a sufficient level of flexibility to experiment with existing and novel approaches to advance the state-of-the-art.
FLIX: A Simple and Communication-Efficient Alternative to Local Methods in Federated Learning
A persistent problem in federated learning is that it is not clear what the optimization objective should be: the standard average risk minimization of supervised learning is inadequate in handling several major constraints specific to federated learning, such as communication adaptivity and personalization control.
Hyperparameter Transfer Learning with Adaptive Complexity
Bayesian optimization (BO) is a sample efficient approach to automatically tune the hyperparameters of machine learning models.
Lower Bounds and Optimal Algorithms for Personalized Federated Learning
Our first contribution is establishing the first lower bounds for this formulation, for both the communication complexity and the local oracle complexity.
A Better Alternative to Error Feedback for Communication-Efficient Distributed Learning
EF remains the only known technique that can deal with the error induced by contractive compressors which are not unbiased, such as Top-$K$.
On Biased Compression for Distributed Learning
In the last few years, various communication compression techniques have emerged as an indispensable tool helping to alleviate the communication bottleneck in distributed learning.
Adaptivity of Stochastic Gradient Methods for Nonconvex Optimization
Adaptivity is an important yet under-studied property in modern optimization theory.
Learning to Optimize via Dual space Preconditioning
Preconditioning an minimization algorithm improve its convergence and can lead to a minimizer in one iteration in some extreme cases.
