Search Results for author:
Found 53 papers, 14 papers with code
Initialization Matters: Unraveling the Impact of Pre-Training on Federated Learning
Several recent works explore the benefits of pre-trained initialization in a federated learning (FL) setting, where the downstream training is performed at the edge clients with heterogeneous data distribution.
Optimized Tradeoffs for Private Prediction with Majority Ensembling
We study a classical problem in private prediction, the problem of computing an $(m\epsilon, \delta)$-differentially private majority of $K$ $(\epsilon, \Delta)$-differentially private algorithms for $1 \leq m \leq K$ and $1 > \delta \geq \Delta \geq 0$.
Federated Communication-Efficient Multi-Objective Optimization
We study a federated version of multi-objective optimization (MOO), where a single model is trained to optimize multiple objective functions.
Nonlinear Stochastic Gradient Descent and Heavy-tailed Noise: A Unified Framework and High-probability Guarantees
While the rate exponents in state-of-the-art depend on noise moments and vanish as $p \rightarrow 1$, our exponents are constant and strictly better whenever $p < 6/5$ for non-convex and $p < 8/7$ for strongly convex costs.
FedECADO: A Dynamical System Model of Federated Learning
Federated learning harnesses the power of distributed optimization to train a unified machine learning model across separate clients.
Debiasing Federated Learning with Correlated Client Participation
In cross-device federated learning (FL) with millions of mobile clients, only a small subset of clients participate in training in every communication round, and Federated Averaging (FedAvg) is the most popular algorithm in practice.
Erasure Coded Neural Network Inference via Fisher Averaging
Erasure-coded computing has been successfully used in cloud systems to reduce tail latency caused by factors such as straggling servers and heterogeneous traffic variations.
FedAST: Federated Asynchronous Simultaneous Training
Much of the existing work in FL focuses on efficiently learning a model for a single task.
FedFisher: Leveraging Fisher Information for One-Shot Federated Learning
Standard federated learning (FL) algorithms typically require multiple rounds of communication between the server and the clients, which has several drawbacks, including requiring constant network connectivity, repeated investment of computational resources, and susceptibility to privacy attacks.
Federated Offline Reinforcement Learning: Collaborative Single-Policy Coverage Suffices
Our sample complexity analysis reveals that, with appropriately chosen parameters and synchronization schedules, FedLCB-Q achieves linear speedup in terms of the number of agents without requiring high-quality datasets at individual agents, as long as the local datasets collectively cover the state-action space visited by the optimal policy, highlighting the power of collaboration in the federated setting.
Efficient Reinforcement Learning for Routing Jobs in Heterogeneous Queueing Systems
Unlike homogeneous systems, a threshold policy, that routes jobs to the slow server(s) when the queue length exceeds a certain threshold, is known to be optimal for the one-fast-one-slow two-server system.
Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Models
Foundation models (FMs) adapt well to specific domains or tasks with fine-tuning, and federated learning (FL) enables the potential for privacy-preserving fine-tuning of the FMs with on-device local data.
High-probability Convergence Bounds for Nonlinear Stochastic Gradient Descent Under Heavy-tailed Noise
First, for non-convex costs and component-wise nonlinearities, we establish a convergence rate arbitrarily close to $\mathcal{O}\left(t^{-\frac{1}{4}}\right)$, whose exponent is independent of noise and problem parameters.
Local or Global: Selective Knowledge Assimilation for Federated Learning with Limited Labels
For both cross-device and cross-silo settings, we show that FedLabel outperforms other semi-supervised FL baselines by $8$-$24\%$, and even outperforms standard fully supervised FL baselines ($100\%$ labeled data) with only $5$-$20\%$ of labeled data.
The Blessing of Heterogeneity in Federated Q-Learning: Linear Speedup and Beyond
When the data used for reinforcement learning (RL) are collected by multiple agents in a distributed manner, federated versions of RL algorithms allow collaborative learning without the need for agents to share their local data.
Federated Minimax Optimization with Client Heterogeneity
We analyze the convergence of the proposed algorithm for classes of nonconvex-concave and nonconvex-nonconcave functions and characterize the impact of heterogeneous client data, partial client participation, and heterogeneous local computations.
On the Convergence of Federated Averaging with Cyclic Client Participation
Federated Averaging (FedAvg) and its variants are the most popular optimization algorithms in federated learning (FL).
FedExP: Speeding Up Federated Averaging via Extrapolation
Federated Averaging (FedAvg) remains the most popular algorithm for Federated Learning (FL) optimization due to its simple implementation, stateless nature, and privacy guarantees combined with secure aggregation.
FedVARP: Tackling the Variance Due to Partial Client Participation in Federated Learning
To remedy this, we propose FedVARP, a novel variance reduction algorithm applied at the server that eliminates error due to partial client participation.
Multi-Model Federated Learning with Provable Guarantees
Federated Learning (FL) is a variant of distributed learning where edge devices collaborate to learn a model without sharing their data with the central server or each other.
Federated Stochastic Approximation under Markov Noise and Heterogeneity: Applications in Reinforcement Learning
Federated reinforcement learning is a framework in which $N$ agents collaboratively learn a global model, without sharing their individual data and policies.
On the Unreasonable Effectiveness of Federated Averaging with Heterogeneous Data
Motivated by this observation, we propose a new quantity, average drift at optimum, to measure the effects of data heterogeneity, and explicitly use it to present a new theoretical analysis of FedAvg.
Federated Learning under Distributed Concept Drift
Federated Learning (FL) under distributed concept drift is a largely unexplored area.
Maximizing Global Model Appeal in Federated Learning
We provide convergence guarantees for MaxFL and show that MaxFL achieves a $22$-$40\%$ and $18$-$50\%$ test accuracy improvement for the training clients and unseen clients respectively, compared to a wide range of FL modeling approaches, including those that tackle data heterogeneity, aim to incentivize clients, and learn personalized or fair models.
Heterogeneous Ensemble Knowledge Transfer for Training Large Models in Federated Learning
In this work, we propose a novel ensemble knowledge transfer method named Fed-ET in which small models (different in architecture) are trained on clients, and used to train a larger model at the server.
Federated Minimax Optimization: Improved Convergence Analyses and Algorithms
In this paper, we consider nonconvex minimax optimization, which is gaining prominence in many modern machine learning applications such as GANs.
FedLite: A Scalable Approach for Federated Learning on Resource-constrained Clients
In classical federated learning, the clients contribute to the overall training by communicating local updates for the underlying model on their private data to a coordinating server.
Leveraging Spatial and Temporal Correlations in Sparsified Mean Estimation
We study the problem of estimating at a central server the mean of a set of vectors distributed across several nodes (one vector per node).
Personalized Federated Learning for Heterogeneous Clients with Clustered Knowledge Transfer
Personalized federated learning (FL) aims to train model(s) that can perform well for individual clients that are highly data and system heterogeneous.
Best-Arm Identification in Correlated Multi-Armed Bandits
In this paper we consider the problem of best-arm identification in multi-armed bandits in the fixed confidence setting, where the goal is to identify, with probability $1-\delta$ for some $\delta>0$, the arm with the highest mean reward in minimum possible samples from the set of arms $\mathcal{K}$.
A Field Guide to Federated Optimization
Federated learning and analytics are a distributed approach for collaboratively learning models (or statistics) from decentralized data, motivated by and designed for privacy protection.
Job Dispatching Policies for Queueing Systems with Unknown Service Rates
In multi-server queueing systems where there is no central queue holding all incoming jobs, job dispatching policies are used to assign incoming jobs to the queue at one of the servers.
Local Adaptivity in Federated Learning: Convergence and Consistency
Popular optimization algorithms of FL use vanilla (stochastic) gradient descent for both local updates at clients and global updates at the aggregating server.
Adaptive Quantization of Model Updates for Communication-Efficient Federated Learning
Communication of model updates between client nodes and the central aggregating server is a major bottleneck in federated learning, especially in bandwidth-limited settings and high-dimensional models.
Bandit-based Communication-Efficient Client Selection Strategies for Federated Learning
Due to communication constraints and intermittent client availability in federated learning, only a subset of clients can participate in each training round.
Client Selection in Federated Learning: Convergence Analysis and Power-of-Choice Selection Strategies
Federated learning is a distributed optimization paradigm that enables a large number of resource-limited client nodes to cooperatively train a model without data sharing.
Probabilistic Neighbourhood Component Analysis: Sample Efficient Uncertainty Estimation in Deep Learning
In this work, we show that the uncertainty estimation capability of state-of-the-art BNNs and Deep Ensemble models degrades significantly when the amount of training data is small.
Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization
In federated optimization, heterogeneity in the clients' local datasets and computation speeds results in large variations in the number of local updates performed by each client in each communication round.
Slow and Stale Gradients Can Win the Race
Distributed Stochastic Gradient Descent (SGD) when run in a synchronous manner, suffers from delays in runtime as it waits for the slowest workers (stragglers).
Machine Learning on Volatile Instances
Due to the massive size of the neural network models and training datasets used in machine learning today, it is imperative to distribute stochastic gradient descent (SGD) by splitting up tasks such as gradient evaluation across multiple worker nodes.
Overlap Local-SGD: An Algorithmic Approach to Hide Communication Delays in Distributed SGD
In this paper, we propose an algorithmic approach named Overlap-Local-SGD (and its momentum variant) to overlap the communication and computation so as to speedup the distributed training procedure.
Advances and Open Problems in Federated Learning
FL embodies the principles of focused data collection and minimization, and can mitigate many of the systemic privacy risks and costs resulting from traditional, centralized machine learning and data science approaches.
Multi-Armed Bandits with Correlated Arms
We consider a multi-armed bandit framework where the rewards obtained by pulling different arms are correlated.
Deep Kernels with Probabilistic Embeddings for Small-Data Learning
Experiments on a variety of datasets show that our approach outperforms the state-of-the-art in GP kernel learning in both supervised and semi-supervised settings.
Accelerating Deep Learning by Focusing on the Biggest Losers
This paper introduces Selective-Backprop, a technique that accelerates the training of deep neural networks (DNNs) by prioritizing examples with high loss at each iteration.
MATCHA: Speeding Up Decentralized SGD via Matching Decomposition Sampling
This paper studies the problem of error-runtime trade-off, typically encountered in decentralized training based on stochastic gradient descent (SGD) using a given network.
MLSys: The New Frontier of Machine Learning Systems
Machine learning (ML) techniques are enjoying rapidly increasing adoption.
Adaptive Communication Strategies to Achieve the Best Error-Runtime Trade-off in Local-Update SGD
Large-scale machine learning training, in particular distributed stochastic gradient descent, needs to be robust to inherent system variability such as node straggling and random communication delays.
A Unified Approach to Translate Classical Bandit Algorithms to the Structured Bandit Setting
We consider a finite-armed structured bandit problem in which mean rewards of different arms are known functions of a common hidden parameter $\theta^*$.
Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms
Communication-efficient SGD algorithms, which allow nodes to perform local updates and periodically synchronize local models, are highly effective in improving the speed and scalability of distributed SGD.
Correlated Multi-armed Bandits with a Latent Random Source
As a result, there are regimes where our algorithm achieves a $\mathcal{O}(1)$ regret as opposed to the typical logarithmic regret scaling of multi-armed bandit algorithms.
Active Distribution Learning from Indirect Samples
At each time step, we choose one of the possible $K$ functions, $g_1, \ldots, g_K$ and observe the corresponding sample $g_i(X)$.
Slow and Stale Gradients Can Win the Race: Error-Runtime Trade-offs in Distributed SGD
Distributed Stochastic Gradient Descent (SGD) when run in a synchronous manner, suffers from delays in waiting for the slowest learners (stragglers).
