Search Results for author:
Found 28 papers, 12 papers with code
Fine-Tuning Large Language Models with User-Level Differential Privacy
We investigate practical and scalable algorithms for training large language models (LLMs) with user-level differential privacy (DP) in order to provably safeguard all the examples contributed by each user.
DrJAX: Scalable and Differentiable MapReduce Primitives in JAX
We present DrJAX, a JAX-based library designed to support large-scale distributed and parallel machine learning algorithms that use MapReduce-style operations.
Leveraging Function Space Aggregation for Federated Learning at Scale
Many federated learning algorithms, including the canonical Federated Averaging (FedAvg), take a direct (possibly weighted) average of the client parameter updates, motivated by results in distributed optimization.
Towards Federated Foundation Models: Scalable Dataset Pipelines for Group-Structured Learning
Finally, it is framework-agnostic.
Gradient Descent with Linearly Correlated Noise: Theory and Applications to Differential Privacy
We propose a simplified setting that distills key facets of these methods and isolates the impact of linearly correlated noise.
Federated Automatic Differentiation
We propose a federated automatic differentiation (FAD) framework that 1) enables computing derivatives of functions involving client and server computation as well as communication between them and 2) operates in a manner compatible with existing federated technology.
Federated Select: A Primitive for Communication- and Memory-Efficient Federated Learning
In order to make this practical, we outline a primitive, federated select, which enables client-specific selection in realistic FL systems.
Motley: Benchmarking Heterogeneity and Personalization in Federated Learning
To better answer these questions, we propose Motley, a benchmark for personalized federated learning.
Optimizing the Communication-Accuracy Trade-off in Federated Learning with Rate-Distortion Theory
A significant bottleneck in federated learning (FL) is the network communication cost of sending model updates from client devices to the central server.
Iterated Vector Fields and Conservatism, with Applications to Federated Learning
In the context of federated learning, we show that when clients have loss functions whose gradients satisfy this condition, federated averaging is equivalent to gradient descent on a surrogate loss function.
A Field Guide to Federated Optimization
Federated learning and analytics are a distributed approach for collaboratively learning models (or statistics) from decentralized data, motivated by and designed for privacy protection.
On Large-Cohort Training for Federated Learning
Our work highlights a number of challenges stemming from the use of larger cohorts.
Local Adaptivity in Federated Learning: Convergence and Consistency
Popular optimization algorithms of FL use vanilla (stochastic) gradient descent for both local updates at clients and global updates at the aggregating server.
Convergence and Accuracy Trade-Offs in Federated Learning and Meta-Learning
Using these insights, we are able to compare local update methods based on their convergence/accuracy trade-off, not just their convergence to critical points of the empirical loss.
On the Outsized Importance of Learning Rates in Local Update Methods
We study a family of algorithms, which we refer to as local update methods, that generalize many federated learning and meta-learning algorithms.
Adaptive Federated Optimization
Federated learning is a distributed machine learning paradigm in which a large number of clients coordinate with a central server to learn a model without sharing their own training data.
Advances and Open Problems in Federated Learning
FL embodies the principles of focused data collection and minimization, and can mitigate many of the systemic privacy risks and costs resulting from traditional, centralized machine learning and data science approaches.
DETOX: A Redundancy-based Framework for Faster and More Robust Gradient Aggregation
In this work, we present DETOX, a Byzantine-resilient distributed training framework that combines algorithmic redundancy with robust aggregation.
Convergence and Margin of Adversarial Training on Separable Data
Our results are derived by showing that adversarial training with gradient updates minimizes a robust version of the empirical risk at a $\mathcal{O}(\ln(t)^2/t)$ rate, despite non-smoothness.
Does Data Augmentation Lead to Positive Margin?
Data augmentation (DA) is commonly used during model training, as it significantly improves test error and model robustness.
ErasureHead: Distributed Gradient Descent without Delays Using Approximate Gradient Coding
We present ErasureHead, a new approach for distributed gradient descent (GD) that mitigates system delays by employing approximate gradient coding.
A Geometric Perspective on the Transferability of Adversarial Directions
We show that these "transferable adversarial directions" are guaranteed to exist for linear separators of a given set, and will exist with high probability for linear classifiers trained on independent sets drawn from the same distribution.
ATOMO: Communication-efficient Learning via Atomic Sparsification
We present ATOMO, a general framework for atomic sparsification of stochastic gradients.
Gradient Coding via the Stochastic Block Model
Gradient descent and its many variants, including mini-batch stochastic gradient descent, form the algorithmic foundation of modern large-scale machine learning.
DRACO: Byzantine-resilient Distributed Training via Redundant Gradients
Distributed model training is vulnerable to byzantine system failures and adversarial compute nodes, i. e., nodes that use malicious updates to corrupt the global model stored at a parameter server (PS).
Approximate Gradient Coding via Sparse Random Graphs
Distributed algorithms are often beset by the straggler effect, where the slowest compute nodes in the system dictate the overall running time.
Stability and Generalization of Learning Algorithms that Converge to Global Optima
Finally, we show that although our results imply comparable stability for SGD and GD in the PL setting, there exist simple neural networks with multiple local minima where SGD is stable but GD is not.
Subspace Clustering with Missing and Corrupted Data
Given full or partial information about a collection of points that lie close to a union of several subspaces, subspace clustering refers to the process of clustering the points according to their subspace and identifying the subspaces.
